Katingan Project, Indonesia / Project Evaluation using Causal Inference

This article is an automatically translated version of the original Japanese article. Please refer to the Japanese version for the most accurate information.

This is sustainacraft Inc.'s 3rd Newsletter. This time, we bring you topics related to forest Carbon Credits.

As introduced in our first Newsletter, numerous important guidelines regarding the utilization of Carbon Credits have been published in recent months. These aim to enhance Credit reliability and promote the circulation of high-quality Credits.

One of the reasons behind these guidelines is the growing criticism in recent years against "junk carbon credits" (Credits issued for nature conservation activities that have no substantial effect as climate change measures).

This time, we introduce some news related to junk Carbon Credits and a method for evaluating carbon projects using causal inference technology.

PickUp Section

(1) The Katingan Project in Indonesia

Nikkei Asia's Criticism of Forest-based Carbon Credits in Indonesia

(Source, 2021/12/13)

• The Katingan Project, aiming to protect forests in Central Kalimantan, Indonesia, is one of the world's largest carbon projects, started in late 2010.

  

• However, half a year after the Katingan Project began, a policy called the forest moratorium was implemented in Indonesia, prohibiting new forest logging and development in a wide area, including approximately two-thirds of the Katingan project area.

• Carbon Credits are calculated based on the difference between the level of Deforestation achieved by the project's implementation (Additionality) and the level of Deforestation that would have occurred if the project had not existed (which is called the Baseline).

• Therefore, even if the Katingan Project had not existed, forest logging would have been suppressed by the moratorium's implementation. Thus, the project's Additionality in areas overlapping with the moratorium could be limited.

• In fact, the authors conducted an independent analysis using satellite imagery and found that in other areas where the moratorium was in effect, forests decreased by "only about 1% annually."

  Using satellite data from the University of Maryland, Nikkei analyzed changes in the forest area in Central Kalimantan and found that except for a year of large-scale fires, forests without concessions decreased by only about 1% a year.

• Despite this, the Project Proponent issued Credits according to the original scenario without revising the Baseline to account for the moratorium. This article claims that Credits were over-issued by up to approximately three times compared to the project's actual effect.

Permian Global's Rebuttal to Nikkei Asia's Article

(Source, 2021/12/21)

• On the other hand, there is a rebuttal to Nikkei Asia's article. Here, we introduce an article by Permian Global, an investment firm active in environmental conservation and one of the Katingan Project's Technical Partners1.

• Nikkei Asia's article argued that "the risk of forest logging was eliminated by the moratorium policy, so Credits issued in areas where the moratorium and project overlapped are not valid." However, this article claims that this is incorrect.

  “possibility of plantation development has been almost fully eliminated by the moratorium. Now, one-third of the project’s anti-deforestation efforts are redundant.” This statement is completely untrue and recklessly naïve.

• Several arguments are presented, but the following two points are particularly important:

  • The moratorium policy stops new permits for forest logging, but concessions already granted permits may continue.

  • Credits are calculated based on the difference in carbon Emission Reductions (Additionality) against the Baseline that would have occurred if the project had not existed, not based on the amount of carbon absorbed/sequestered by the target forest itself.

    The moratorium policy, which was announced later, and which applied only to new permits, would not have affected those concessions already granted with a permit, so they would have been able to continue operating.

    Credits are issued based on the avoided emissions generated by the project through its elimination of threats. Credits are not generated by the amount of carbon dioxide absorbed / sequestered.

• Therefore, it is stated that judging a project's ineffectiveness by monitoring other forests where the moratorium is implemented using satellite imagery and confirming a low Deforestation rate, as done in Nikkei Asia's article, is incorrect.

(2) Analyzing the Effectiveness of Forest Projects Using Causal Inference Methods

Setting the Baseline in Forest Projects

• One of the factors causing the disagreement in the two articles above is the issue of Baseline setting.
• This is due to differing perceptions of external environmental changes, such as the moratorium implemented after the project's start. However, not limited to this example, Baseline setting inherently involves much Uncertainty because it targets something that cannot be observed.
• In VM0007 / VM0015, one of the main REDD+ Methodologies, the Baseline is set by extrapolating past Deforestation trends (at least 10 years) in the areas surrounding the project area. Naturally, this method does not reflect the impact of policy changes that occurred after the project application.
• Furthermore, overestimating the Baseline Deforestation can lead to an increase in Issued Credits, potentially introducing bias from the applicant.
• Therefore, to generate reliable Credits, a Methodology is needed that objectively evaluates the direct effects of a project (Additionality) by excluding the influence of external factors as much as possible.
• As an example of such an attempt, we introduce a study below that re-evaluated carbon projects using a statistical project evaluation method called Causal Inference, specifically a model known as Synthetic Control Method.

West et al. (2020): Re-evaluating REDD+ Projects in the Amazon Region Using Causal Inference Methods

(Source: West et al., 2020)

• This paper re-evaluates 12 REDD+ projects conducted in the Amazon region using the Synthetic Control Method (SCM, Abadie et al., 20102), one of the Causal Inference methods.

• SCM is a method that evaluates the time-series effect of a project by appropriately weighting data from multiple regions where no project was implemented to statistically estimate a "hypothetical Deforestation trend that would have occurred if the project had not been implemented in that region" (which is precisely the Baseline in Carbon Credits).

• Since the Baseline is calculated using data from reference regions before and after the project's implementation, any external factors such as policy changes in the reference regions can be reflected in the generated Baseline. This allows for estimating only the project's pure effect.

• In the Amazon region, which is the subject of the titled paper, similar to the Katingan case, a forest logging suppression policy (National Climate Change Plan) was implemented by the Brazilian government after the project's start. Therefore, the presence or absence of project Additionality is a key issue. This is verified using SCM.

• As a result, it was concluded that many Credits had been Issued despite almost all projects showing no statistical significance (= the null hypothesis that the project's effect is zero could not be rejected) (X-axis is time, Y-axis is change in Deforestation. The further the red line swings downwards, the greater the project's effect3).

  

• The reason for this is believed to be that the Baseline at the time of project application did not consider external changes such as the implementation of forest logging suppression policies, leading many projects to overestimate future forest logging trends
  (Y-axis is cumulative forest logging, orange is the Baseline at the time of application, blue is the Baseline estimated by SCM)

  

News from sustainacraft

● Selected for SAP.iO Foundry Tokyo

• This is a startup support program implemented by SAP Japan.

Closing remarks

The Baseline presents a fundamental difficulty in that it is an unobservable quantity as data. This is known as the fundamental problem of Causal Inference (Holland, 1986), a challenge that cannot be resolved, no matter how abundant the data.

Below is a slide presented at a recent Science Based Target for Nature session4. While several frameworks for evaluating Natural Capital are emerging, I believe the following conflicting structure also applies to Carbon Credit evaluation.

While our company conducts evaluations within a unified framework, we also intend to carefully interpret data and disclose information regarding project-specific elements that fall outside that framework.

This concludes sustainacraft's Newsletter #3. In this Newsletter, we plan to disseminate information in Japanese on NbS roughly bi-weekly to once a month.

Our company profile materials are available here for your reference.

Disclaimers:
This newsletter is not financial advice. So do your own research and due diligence.