Counting regulations and measuring regulatory impact: a call for nuance

Abstract

The effect of regulation on virtually every aspect of the lives of US citizens has led to an understandable impulse to measure this total impact. It has led to various attempts to count the total number of regulations and regulatory requirements, and to total the costs and benefits of regulation. And these counting mechanisms have played prominent roles in discussions over statutory changes designed to reform the process by which we write regulations. However, counting regulations in a meaningful way and measuring their cumulative economic impact is an astonishingly difficult task. For this reason, there have been a wide variety of methods that scholars and advocates have employed in the effort to do so. This article is an attempt to catalog the most prominent methods of counting regulations and measuring regulatory impact in the United States, describe their strengths and weaknesses, and suggest alternative approaches to attack this important question. We suggest both using large language models and detailed analysis of Paperwork Reduction Act data and, at the opposite extreme, doing more qualitative work on the consequences of regulation on individuals, firms, and industries.

Introduction

By any measure, the regulatory state in the United States has expanded tremendously in the nearly 80 years since the passage of the Administrative Procedure Act. This expansion has led to cleaner air, safer workplaces, and many other benefits. At the same time, the regulations imposed over the past eight decades have placed a great deal of burden and costs upon regulated industries which, in addition to resulting in reduced profits, has likely also resulted in higher prices for consumers and job losses for some workers (Dudley and Mannix, 2018).

The impact of regulation on virtually every aspect of the lives of US citizens has led to an understandable impulse to measure this total impact. Much of this debate has played out in the context of discussions of the role that cost–benefit analysis should play in regulatory decision-making (see e.g., Arrow et al. 1996; Ackerman and Heinzerling, 2001). But it has also played out in various attempts to count the total number of regulations and regulatory requirements, and to total the costs of regulation (Carey, 2019). And these counting mechanisms have played prominent roles in discussions over statutory changes designed to reform the process by which we promulgate regulations (Short, 2018).

However, counting regulations in a meaningful way and measuring their cumulative economic impact are both very difficult tasks. For this reason, there have been a wide variety of methods that scholars and advocates have employed in the effort to do so. This article is an attempt to catalog the most prominent methods of counting regulations and regulatory impact, describe their strengths and weaknesses, and suggest alternative approaches to attack this important question. By having all of these methods discussed in one place, I hope to both shed light on the counting enterprise and posit lessons for better measuring the overall impact of the regulatory state.

For reasons of space and tractability, this article focuses nearly exclusively on the efforts to count regulations in the United States. There have been many efforts, including many that are using large language models, to count regulations in Europe. Where there are clear lessons for the US counting enterprise, I will note these below. However, the efforts at counting regulations and measuring regulatory impact in the United States have been much more directed at the question of “Are there too many regulations?” than in Europe. It is those efforts that this article is aimed at addressing.

This article is divided into four broad parts. The first is a review of the arguments for and against counting regulations at all and measuring regulatory impact. The second focuses on the various methods of actually counting the number of US regulations. The next section discusses measures of the total impact of US regulation. The final section offers concluding thoughts on both the need for counting and the need to do it better.

Why count?

In the US, regulations are generally understood to be policy prescriptions issued by agencies in the executive branch of government or by independent commissions pursuant to statutes passed by Congress. These regulations are also understood to produce social benefits (often improved health and safety) and to impose costs upon the regulated community. They may also set the criteria for receiving benefits from the government (Kerwin et al. 2018). Attempts at counting or measuring impact have largely focused on federal regulations, but there are also many regulations issued by state, local and Tribal governments.

It is relatively undisputed that most efforts to count regulations or to total the impact of regulations come from institutions or individuals who favor a reduction in the role that government regulation plays in everyday life. This has inevitably shaped the debate over counting regulations and measuring total regulatory impact. One of the critical studies was written for the National Association of Manufacturers (Crain and Crain, 2014). Earlier versions of the report were written for the Small Business Administration’s Office of Advocacy, which regularly opposes regulations that impose burdens on small businesses (Carey, 2016). Crews’ (2021) “Ten Thousand Commandments” an annual count of federal regulations, describes its mission as “to reform America’s unaccountable regulatory state.”¹ The Mercatus Center which is also discussed further below is less direct but their Facebook page describes them as a “research center that focuses on how markets solve problems.”²

The central argument made by all of these studies is that the volume of regulation and the cost of regulation collectively stifle business innovation and reduce the nation’s economic productivity. Even if individual regulations may make sense (or have net societal benefits), collectively, the volume of federal regulations is a significant problem for American businesses. Others have argued that regulatory accumulation also increases the chances of undesirable interactions between regulations and of “behavioral overload” which makes it hard for individual managers to focus on anything beyond rote compliance (Mandel and Carew, 2013).

In part, because counts of regulation are largely done by those opposing government intervention in the market, regulatory supporters have taken to criticizing the enterprise of regulatory counting. They have made broad theoretical arguments against counting, citing its oversimplification of complex phenomena. They also make more practical arguments, saying counting undermines statutory goals, is wasteful, and crowds out a meaningful dialog about regulatory policy (Short, 2018).

It is unclear how private academics or institutions conducting counts of regulation or measures of regulatory impact undermine statutory goals unless they lead to policies like the “two for one policy” put forward by the Trump Administration in Executive Order 13771.³ But if done well, there is no reason that bad policy must flow from counting regulation or measuring regulatory impact. To the extent that counts are undermining agency morale or dampening regulatory output, that is probably because the counts are done poorly and with an intent to argue against agency action.

Typically, very little government effort is spent in counting (besides by the Office of Management and Budget totaling up costs and benefits calculated by regulatory agencies as described below), and if private entities want to count, it is hard to categorize that effort as “wasteful.” Having individual agencies calculate the costs and benefits of their individual regulations may be wasteful, at least as seen by supporters of those regulations (McGarity, 1992), but that is a separate question from the larger counting enterprise that it feeds into (Dudley et al. 2017).

There are, however, reasons to better understand the cumulative impact of regulations on individuals and on businesses. The first is that it is possible that it matters in the sense that critics of regulation assert; that too much regulation may dampen economic productivity and innovation (Crain and Crain, 2014). But while this is a contested assertion, it is also true that duplicative requirements (not all coming from the federal government, but also from state and local governments⁴) and obligations whose purpose is obscure to those who follow the rules reduce faith that the government “knows what it is doing,” Regulations are a burden to businesses and to individuals. Whether those burdens are justified is a separate question. (Shapiro and Borie-Holtz, 2020; Jones and Graf, 2001). Regulations are also a source of “red tape,” requirements that require compliance but serve an ambiguous (at best) purpose (Bozeman and Feeney, 2014).

A second reason to devote meaningful attention to measuring regulatory impact is that principled opposition to counting regulations is not going to stop such efforts by those who hope that such counts will mobilize opposition to the regulatory state. It will merely leave the field open to such efforts. The argument against flawed methods for counting regulations or measuring regulatory impact should certainly include highlighting the flaws with these methods. But to promote an honest debate about regulatory impact, such arguments should also attempt to imbue regulatory counts with meaning and continually improve upon them.

Relatedly, in public debates, numbers have power (Porter, 1996). In a debate over the role of government regulation, if one side is arguing by citing large volumes of requirements or the burden (but not the benefits) of those requirements, even if (or especially if) those numbers are inflated, they are going to have an inherent advantage in that debate. Arguments supporting regulatory agencies are inevitably strengthened if advocates can either effectively point out that burdens are not as large as portrayed, or that tallies of regulatory benefits, including lives saved, illnesses averted, or reduced risks justify the burdens imposed. Both of these techniques require active engagement in discussions of the cumulative impact of regulations.

Counts of regulations and regulatory impact allow us to better understand the effect of changes in the regulatory process or in the political climate. To take one example, it was only by counting the regulatory and deregulatory actions under the Trump Administration that scholars were able to conclude that claims about deregulation during that presidency were vastly exaggerated and largely incorrect (Coglianese, Sarin et al. 2021). Counts of regulations have also been used to highlight the tendency during presidential administrations to issue more regulations as they approach their conclusion (i.e. “midnight regulations”) (Brito and De Rugy, 2009). Measures of costs and benefits have been used by many scholars to draw conclusions about presidential priorities and those president’s success in achieving them (Masur, 2019).

Finally, cumulative regulation and regulatory burden do not just affect businesses. Individuals, particularly those entitled to government benefits, are subject to a bevy of regulatory requirements in order to secure those benefits (Herd and Moynihan, 2019). The nature of this burden differs in that businesses are forced to comply with regulations that largely benefit others, and many individuals must fulfill burdensome requirements to obtain government entitlements. But much like the costs faced by businesses, those faced by individuals are real, potentially large cumulatively, and worth being a part of any discussion of regulatory costs. In addition, these individuals may be the most unable to cope with the cumulative regulatory burden because of the many other factors demanding their attention (Mullainathan and Shafir, 2013), and hence, the burden may deter them from applying for government benefits to, which they are entitled. The federal government has recently devoted attention to reducing these impacts (OMB, 2023c), but understanding the total magnitude of these effects would dramatically assist these efforts.

These are all reasons to measure regulatory impact (including regulatory benefits). But to realize the benefits of measurement as described in the above paragraphs, any efforts, including counting regulations, need to be done well. And many methods of counting regulations to date have been fundamentally flawed. Their proliferation has also led to considerable confusion regarding the nature of regulatory impact in the United States. In the next two sections, I chronicle the various attempts to count regulations and measure cumulative regulatory impact, highlighting each measure’s strengths and (usually greater) weaknesses. I hope to address the question of how current attempts in the US to count regulations and measure impact are flawed and what would better methods look like.

Counting rules

Counting Federal Register pages and documents

The simplest approach to highlight the reach of the regulatory state is by counting regulations in the Federal Register.⁵ Agencies are required to publish their regulations, including proposals and supplemental documents, in the Federal Register. Therefore, it is a natural temptation to count pages and documents. At the very least, shouldn’t changes in these counts reflect changes in regulatory impact?

As H.L. Mencken said, “For every complex problem, there is an answer that is clear, simple, and wrong.”⁶ There are a number of obvious problems with counting Federal Register pages or documents. A recent report by the Congressional Research Service describes these problems. Counting Federal Register pages is a vast overstatement of the extent of regulation, because the Federal Register publishes many other documents such as meeting notices, required notices under the Paperwork Reduction Act, proposed regulations and even blank pages. In addition, many of the pages devoted to final rules are the preamble to those rules which explain their content and contain agency responses to public comments (Carey, 2019).

Counting the number of final rule documents in the Federal Register is an improvement but still presents numerous challenges. Most importantly, such a count does not distinguish between types of rules. Rules that are routine and have little impact on the public are counted the same as those which are extremely significant.⁷ Similarly, a final rule is required to rescind an existing regulation meaning that the removal of a regulation counts the same as the issuance of a regulation (Carey, 2019).

The increase in pages and documents in the Federal Register between 1968 and 1980 (Carey, 2019) almost certainly is reflective of the growth in both the volume of regulation and the magnitude of regulatory impact during this period. However, the year-to-year variations since this period are much harder to interpret. And uses of this data to claim that a particular administration or time period was heavily regulatory (or deregulatory) should be greeted with a considerable dose of skepticism.

Counting CFR pages

The idea of counting pages in the Code of Federal Regulations (CFR)⁸ is in many ways an improvement over counting Federal Register pages or documents. The CFR includes only regulations, not the meeting notices or other non-regulatory documents that fill the pages of the Federal Register. As one author argues,

The CFR contains literally every federal regulation in existence during a given year, and it has a time span of more than 50 years…. Because all federal regulations must be published in the CFR, our page count measure must have at least a rough correlation with the “true” amount of regulation that should enter an economic model. If the CFR page count were zero, there would be no regulation, and it surely is reasonable to suppose that the more pages there are in the CFR, the more regulations there are. (Dawson and Seater, 2013).

Despite these attractions, there are a number of challenges with using CFR pages to measure the reach of regulation. One is that it never goes down. Even in administrations ostensibly (Coglianese, Sarin et al. 2021) dedicated to deregulation, the number of CFR pages holds steady or increases with each passing year (Crews, 2021). This makes it hard to interpret growth in CFR pages. The rate of growth may be useful, and some analysts have used this (Dawson and Seater, 2013), but there are also some fundamental challenges with this metric as well.

Many of these are outlined by Professor Jodi Short (2018). Professor Short describes a series of problems with counting CFR pages or sections. Among the most significant ones are the fact that different requirements impose differential levels of burden, different requirements apply to differing universes of regulated entities, page counts include restrictions on government entities as well as private actors, and many regulations contain exceptions (see also Coglianese et al. 2021), options for regulated entities, and references to other regulations as well as numerous other stipulations that do not impact regulated entities.

Counting regulatory restrictions

The challenges posed by the many components of non-regulatory language in the CFR have been made easier to overcome by advances in technology. Theoretically it should be possible to take out the non-regulatory portions of the CFR and count the requirements that remain. The desire to actually count the number of regulatory restrictions in the CFR led Patrick McLaughlin and a number of other scholars at the Mercatus Institute to create RegData. As described in a series of articles (Al Ubaydli and McLaughlin, 2017; McLaughlin and Sherouse, 2019), this database has the goal of combating many of the problems associated with using the number of pages in the CFR or the number of documents in the Federal Register. Reg Data uses text analysis to count the number of “binding constraints”⁹ in each chapter of the CFR. Their work shows a continual increase in these words over a 40-year period.

Beyond merely counting the incidence of these words and phrases, RegData also uses North American Industry Classification System (NAICS) codes to break up the CFR according to which chapters affect which industries. The database has been updated several times and now includes information on which agencies promulgated regulations and the laws pursuant to which particular regulations were issued (McLaughlin and Sherouse, 2019).

The appeal of this approaches¹⁰ is obvious as it appears to be more calibrated than the word counts and document counts described above. And there is little doubt that it is an improvement on the methods described above. However, many of Professor Short’s criticisms of those counts also apply to the RegData methodology. For example, she cites a passage in the CFR that contains the word “require” five times but only represents one requirement. She also notes that often words counted by RegData as requirements have negations modifying them. (i.e. “shall not require”). Finally, she notes that some language flagged in the RegData database appears in the CFR as clarifying questions, (“Must I sign a service agreement” where “must” would be flagged as a requirement). This collection of concerns raises significant questions about the use of this database to give a picture of regulatory impact as well as highlighting the challenges associated with using new sophisticated AI tools to measure regulatory impacts (Short, 2018).

Using large language models has also been a feature of recent work measuring regulatory impact in the European Union (Franchino et al. 2024). Studies here, though, have also found that “various measures capturing policy substance and the structural characteristics of legal acts context are not systematically related and capture different empirical phenomena,” (Hurka et al. 2024, p. 725). Also, there has not yet been a way devised to use these models to analyze the benefits of regulation.

Counting documents reviewed by OMB

The desire to use page counts or more sophisticated measures of restrictions is understandable. It translates a complicated question into a simple number that hopefully conveys the impact of a growing volume of regulation. But as the previous sections make clear, there are many problems with using these simple counts as measures of the impact of regulation. One alternative approach is to simplify the task of counting in a different manner. One could, instead of counting all regulations, count only those which have a notable impact on public welfare.

Fortunately, there is a relatively easy way to do that. In 1981, President Reagan issued Executive Order 12291, requiring the Office of Information and Regulatory Affairs (OIRA) to review all executive branch agency regulations and requiring agencies to conduct “Regulatory Impact Analyses (RIAs) of the ones with more than $100 million impact in any year. The Reagan EO was modified in 1993 by President Clinton, who replaced it with Executive Order 12866. The new order required that OIRA review only “significant regulations” and gave four categories of such rules. One of the categories mirrored the $100 million threshold of the Reagan Order, and agencies were still required to conduct RIAs for this subset of significant regulations.

This categorization creates two categories of regulations that can be relatively easily tabulated. “Significant” final regulations, those reviewed by OIRA, generally number between 100 and 400 per year (the only exception was 2017, when fewer than 100 were published).¹¹ “Economically significant” final regulations, those that cross the threshold requiring an RIA, have numbered between 15 and 125 per year over the past three decades.

This data is more useful than the mere counts of pages or documents in the Federal Register or CFR. First of all, it includes only regulations likely to be thought of when debates over regulation take place (environmental regulations, immigration regulations, food safety regulations, etc.) while omitting many regulations that are routine or impose little if any burden. Second, with two separate categories it allows researchers to examine both a wide variety of regulations or only those with the largest economic impacts. Third, the data is available in a way that is easy to break up by Cabinet Department or subject area. Finally, a mere glance at it reveals important trends such as the “midnight regulation” phenomena described above (Brito and De Rugy, 2009).

The challenge with using counts of regulations reviewed by OMB as a proxy for regulatory impact is, unlike Federal Register and CFR counts, one of underinclusivity rather than overinclusivity. Regulations issued by independent commissions, those agencies not directly reporting to the President, such as the Securities and Exchange Commission, the Federal Trade Commission, and the Federal Communications Commission, are not subject to OIRA review. Even within the executive branch, the Internal Revenue Service (IRS) also has an exemption from OIRA review that was recently broadened. (OMB, 2023e). Many of these agencies exempt from OIRA review produce a consistently high number of regulations each year, some of which have very significant economic impacts.¹²

In addition, while the OMB review captures many of the regulations from executive branch agencies, some of those omitted because they fail to meet the “significance” threshold in Executive Order 12866 should be counted if we want a true picture of the regulatory burden (Hahn and Cecot, 2007). This challenge may be exacerbated by recent changes made by the Biden Administration to change the definitions of “significant” and “economically significant,”(White House, 2023)

Despite the omissions of the counts of regulation from independent agencies, the IRS, and of nonsignificant regulations, the OMB counts are a superior method of counting regulatory impact to the previous methods discussed.¹³ The Regulatory Studies Center regularly tracks the annual number of significant regulations and economically significant regulations.¹⁴ In addition, the data can be parsed by the agency using databases available on a federal website (www.reginfo.gov). None of this tells you the nature of the regulations being issued, however or the extent or direction of the impact (positive or negative).

Counting Unified Agenda entries

A final source of information on counts of regulations is the Unified Agenda. The Unified Agenda is published twice per year, and each agency that issues regulations lists its planned regulations, proposed regulations, final regulations, and completed regulations.¹⁵ The first three categories get the most attention but are merely predictions or plans. They paint a picture of what an agency intends to publish in the months and year ahead but nothing more than that.

The “completed actions”, however, hold more promise as a count of regulatory actions taken over the previous six months. In a paper measuring the regulatory reach of the Trump Administration, the authors used this metric and found it had several advantages and disadvantages (Coglianese, Sarin et al. 2021). Chief among the advantages is that it is more comprehensive than the OMB regulatory counts, and unlike the Federal Register, it omits non-regulatory documents. Unlike counts of regulations reviewed by OMB, independent agencies publish their list of completed actions. Both “significant” and “non-significant” regulations are included and labeled as such.

In addition, each entry has additional information characterizing the final regulatory action. During the Trump Administration, a field was added to these fields entitled, “EO 13771 Designation,” which was given the categories, “regulatory,” “deregulatory,” “Exempt” “Other,” or “Not subject.” The additional fields in each Unified Agenda entry allow for the parsing of the data into economically significant, significant and non-significant. It also allows breaking the rules up by the agency issuing the regulation, whether a regulation affects small businesses, and in numerous other ways. Collectively, this opens up possibilities for a greater understanding of regulatory impact overall or during a particular time period.

However, the data is not perfect. This was particularly true for the “EO 13771 Designation” field. Actions that should have been classified as regulatory were often put in the three ambiguous categories listed above. And even those actions classified as deregulatory were often the withdrawal of proposed rules rather than actions eliminating actual regulations. The Biden Administration has since eliminated this categorization (having repealed EO 13771), which eliminates the problem of questionable categorization, but brings back the problem of whether any count of regulatory action can meaningfully distinguish between actions that are intended to have a regulatory or deregulatory effect.¹⁶ One other challenge is that data collection is extremely burdensome, but it is possible that AI tools will make it simpler.

For counts of regulatory actions though, the “Completed Actions” section of the Unified Agenda has many advantages over the other methods detailed in this section. It has the best claim for not being either overinclusive or underinclusive. It allows researchers to parse the data in various ways that may be useful. And it is updated every six months. For counts of regulations, it is probably the best measure available.

But the question still persists of what do counts of regulations mean? The concerns raised by critics of the rudimentary methods, counts of Federal Register documents or CFR pages never entirely vanish, even if the sophistication of the methods of counting increases. If we want to understand the impact of regulations on the regulated community (or the benefits of those regulations), none of these measures are terribly helpful. At most, they indicate trends over time or between presidential administrations. What is the difference, though, between 3000 regulations and 4000 regulations? Is there an optimal number of regulations? Even the most sophisticated counts do not get at these questions, and as one author put it, “flawed measures can misrepresent a phenomenon, overestimate or underestimate its severity or diffusion and ultimately undermine the validity of findings” (Damonte and Bazzan, 2024, p. 657). It sounds obvious, but if we want to understand regulatory impacts, we need measures of impact.

Measuring regulatory impact

Ever since the expansion of the regulatory state into areas like environmental protection and worker safety in the 1970s, there have been calls to measure the costs and benefits of regulations (Weidenbaum, 1978). The Ford and Carter Administrations followed through on these calls by creating requirements for the measurement of some regulatory impacts (Tozzi, 2011). As noted above, President Reagan formalized these requirements in Executive Order 12291 issued in 1981. This EO and its successor issued by President Clinton, have formed the basis of most attempts to total up regulatory costs and benefits and understand cumulative regulatory impacts. I divide those efforts below into “bottom up” and “top down” estimates.

Top-down approaches examine the economy as a whole and, using regression analysis attempt to analyze the (assumed to be negative) impact of regulation on the macroeconomic variables such as production, productivity, and employment. The most well-known top-down estimate of regulatory costs is a study originally commissioned by the Small Business Administration authored by Crain and Crain (2014). The Crain study estimates the total cost of regulation as $2.028 trillion with $1.439 trillion coming from their top-down estimate and the rest coming from a traditional bottom-up analysis of regulations (Carey, 2016). The estimate is a large one and has been used frequently by critics of regulation to decry the burden and reach of the regulatory state (Parker, 2019).

But the estimate has also been widely criticized (to the point where it might be better to describe it as debunked). Richard Parker (2019) wrote an article-length critique of the Crain and Crain methodology, which does not need to be repeated here. His key points are that there are serious methodological errors with the regression analysis conducted by the Crains, parts of the Crain’s analysis rely on a World Economic Forum poll that does not deal specifically with regulations, and the analysis includes tax costs that aren’t regulatory in nature. The Small Business Association, which sponsored the first version of the Crain and Crain study, backed away from its conclusions (Sidney Shapiro, 2013a). Later versions of the study were sponsored by the National Association of Manufacturers (NAM, 2023). Because of these flaws, bottom-up measures, for all of the challenges associated with them, are superior to top-down measures produced to date.

Bottom-up approaches to regulatory costs

As described above, Executive Order 12866 requires agencies to calculate the benefits and costs of any regulation that has an impact of more than $100 million per year. These regulations are described as “economically significant regulations.” The same definition was later used for a “major” regulation under the Congressional Review Act. For purposes of totaling up regulatory costs (or benefits), the key question is which regulations have a regulatory impact analysis, which includes a numerical tabulation of costs and benefits. Then, the costs and benefits calculated for individual regulations can be aggregated to come up with an estimate of the total regulatory burden and total regulatory benefits.

In theory, each year’s total should include all regulations for which an estimate of costs and benefits is required under E.O. 12866. OMB is required to submit to Congress an annual report on the total costs and benefits of regulations. This requirement was regularly fulfilled by OMB from 2000 until 2017. The Trump Administration did not issue a final report on the costs and benefits of its own regulations until issuing one report in 2020 which covered the previous three years. The Biden Administration issued its first report in February 2024 and it also covered three years (FY 2020, FY 2021, and FY2022). These reports are the most definitive compilations of required agency cost–benefit analyses.

Each report details the benefits and costs calculated by agencies for their regulations over the past year. The pre-2017 reports also include compilations of the costs and benefits from the previous ten years (essentially adding up the data from the previous year with the data from the data from the prior nine reports). The Trump Administration broke with this tradition, omitting totals from the previous year in the report (OMB, 2017) and instead directing readers to a spreadsheet on their website, in effect forcing readers to do the calculations themselves. The Biden Administration adopted this practice as well (OMB, 2024).

Therefore, the last report with bottom-up estimates of regulatory costs and benefits was the 2017 report issued at the beginning of the Trump Administration. Tables 1–4 from this report are replicated in the Appendix, with rows added for 2017–2022 based on the spreadsheets on the OMB website. Adding these totals for costs and benefits together, for the decade from 2007 until 2022, in 2022 dollars, the calculated benefits of federal regulations ranged from $415.7 billion to $1,277.7 billion. The costs over the same period ranged from $102.9 billion to $163.5 billion.¹⁷ This is probably the best estimate of the costs and benefits of this group of regulations that exists. However, it is still a flawed estimate of cumulative impact.

Most obviously, this estimate is underinclusive. Regulations issued prior to the window are likely still imposing costs and conferring benefits that are not included in these aggregate totals. Independent agencies are not included in the aggregate estimates. For regulations that cross the threshold of requiring an economic analysis, not all of them quantify costs (and many more of them omit estimates of regulatory benefits). The aggregate numbers given by OMB include only those rules for which agencies estimate both benefits and costs (OMB, 2017). And OMB notes, “The monetized estimates we present necessarily exclude unquantified effects (OMB, 2017).”

It is also not as if the regulations that fall below the threshold in Executive Order 12866 for conducting an economic analysis do not impose regulatory burdens or produce benefits. Individually, these rules may each produce fewer than $100 million in annual costs, but collectively (as seen in the imperfect regulatory counts described above), there are a large number of such rules, and their impacts could easily be considerable (Hahn and Cecot, 2007). Finally, as OMB notes, “the most fundamental purpose of a regulatory impact analysis is to inform policy options at the time a regulatory decision is being made; however, analytic approaches that serve this purpose may not readily lend themselves to aggregation (OMB, 2021).”

Using the regulations in the OMB report also leads to the possibility of overinclusivity as well as underinclusivity that mirrors using the counts of regulations reviewed by OMB. Some regulations are overturned in the courts after they are included in OMB reports and, hence, never impose costs or lead to benefits that are included in the totals (Parker, 2019). In addition, the estimates in the OMB report are ex ante estimates rather than ex post ones. There have been analyses of the accuracy of the costs and benefits estimates conducted by agencies under E.O. 12866. While the results vary by regulation, there is a general tendency to overestimate both costs and benefits in individual analyses, which would also lead to an overestimation of both in aggregated estimates (Carey, 2016).

The bottom-up approach of counting regulatory costs is appealing in that it comes from analyses conducted by agency experts. And to the extent that we care about regulatory costs on the economy as a whole, these are likely our best estimates. However, as the discussion above demonstrates, bottom-up aggregate estimates are plagued by inclusion errors. They also obscure the nuance of how the burdens and benefits of regulation are distributed by simply adding up numbers from regulatory agencies with widely varying purviews. The inclusion and accuracy errors may be fixable, but these estimates will always leave out important information about the aggregate impact of regulation on individual actors.

Paperwork burden

One method of assessing regulatory impact that has received less attention in the academic literature than those described above is the measure of the burden of information collection requirements imposed by the government. This omission is curious for a couple of reasons. First the OECD has developed an entire framework, entitled the “Standard Cost Model” (SCM) that focuses on administrative burden (OECD). The model has been adopted by a number of European Union countries.

The second reason is the relative neglect of paperwork burden in the counting of regulatory impact¹⁸ is surprising is that the topic of administrative burden has received a great deal of public attention lately. The book Administrative Burden by Donald Moynihan and Pamela Herd has sparked a great deal of interest in the ways that regulatory requirements deter those eligible for government benefits from applying for them. The Biden Administration took this message to heart when rolling out its student loan forgiveness program and was widely praised for constructing a process for applicants that was simple and widely used (Herd and Moynihan, 2022). If the administrative burden is such a relevant part of the way citizens and firms experience the regulatory state, efforts to count it should be central to assessing the extent of regulatory reach.

Finally, unlike many measures of regulatory impact, there is a (usually) regular and potentially comprehensive calculation of paperwork burden published by the Office of Management and Budget (OMB). Under the Paperwork Reduction Act (PRA), OMB is required to publish annually the Information Collection Budget (ICB). Based on agency submissions under the PRA, which requires agencies to calculate the burden in hours of nearly every information collection conducted or sponsored by the agency, the ICB totals these hours across the federal government. Unfortunately, as with the Annual Report on the Costs and Benefits of Federal Regulations described above, there have been significant gaps in publishing the ICB. After not publishing a report between 2019 and 2022, OMB finally published one in May 2023 (OMB, 2023b).

An examination of the most recently published ICB gives an example of the information gleaned by focusing on the paperwork burden. The Appendix includes an excerpt of Table 1 from that report (OMB, 2023b).

The data in the table reflects a bottom-up approach to counting government-imposed burdens. Agencies imposing the information collection or recordkeeping requirements are charged with calculating the burden estimates, which must also pass through a stage of public comment and review by the Office of Information and Regulatory Affairs. In other words, they are vetted similarly to the estimates of the costs and benefits of regulations and are far more comprehensive.

The numbers are not without their faults, however. Most importantly, if the question is about regulatory burden, paperwork counts are far broader than regulatory counts. They include tax returns (the large number from the Department of the Treasury in Table 2 in the Appendix is almost entirely due to tax burden—and represents significantly more than half the government-wide total burden), surveys such as the US Census, and many other non-regulatory requirements. On the one hand, these burdens do reflect the imposition of requirements from the state. On the other hand, debates about regulatory burden rarely include these types of requirements, and including them could easily obscure debates about regulation (although, theoretically, it should not be hard to remove them).

Also, like estimates of the costs of regulation, estimates of paperwork burden provide a necessarily incomplete picture of the regulatory state, even if we went through the work of removing the non-regulatory burdens from any count. Regulatory information collection and recordkeeping requirements are underinclusive both in that they do not include all regulatory requirements; the requirement to install pollution control equipment would not be counted, but the required documentation of that installment would be, and in that, they (like the cost numbers described above) do not include any quantified measure of regulatory benefits (or “practical utility” in the words of the Paperwork Reduction Act).

Third, also like estimates of the costs and benefits of regulation, estimates of information collection burden are criticized for their lack of accuracy (Shapiro, 2013b). Once calculated, the estimates are almost never meaningfully revisited once the actual burden is imposed. Many agency estimates are based on models that have never been tested empirically. And agencies have the incentive to underestimate the burden of the obligations they impose so as to not appear too heavy-handed.

Finally, while agencies are supposed to justify the burden of their information collection requirements by documenting the “practical utility” of those requirements, this is in practice rarely done and almost never quantified. Hence an assessment using only numbers from the Paperwork Reduction Act will give a better picture on one side of the regulatory ledger but add nothing to the other side. It cannot be an exclusive measure of regulatory impact.

Despite these flaws, the aggregation of burden hours should be a part of any discussion of counting regulatory impact. Clearly, they should not be the only measure for doing so. But they also lend themselves to disaggregration to better understand the requirements imposed by particular offices within agencies and to understand the burden faced by particular industries or groups of individuals. Unlike the other flawed methods described above, the information collected in the Information Collection Budget has not been used in this way.

Suggestions for counting

The impulse for counting regulations and measuring regulatory impact is understandable. As the role of the state has expanded since the early 20th Century, so has the desire to understand the scope and impact of that expansion. Understanding that impact could potentially lead to better design of regulations and greater benefits from them at a lower cost. Indeed, it was that impulse that in part led to movements requiring cost–benefit analysis of regulations (Tozzi, 2011) and the passage of the Paperwork Reduction Act.

However, the efforts at counting aggregate regulatory scope and impact have been flawed and one-sided. There is a simple political economy argument as to why they have been one-sided. Most of those who are motivated to count regulations have done so in the hope that the presentation of large numbers of regulations, high total costs, and large regulatory impacts on the economy will motivate opposition to new regulations or support for the repeal of existing rules. They highlight the growth of regulation and find relationships between regulation and economic conditions that are, in truth, mixed. They present large counts of regulation or regulatory costs without any acknowledgement that those counts don’t speak to the wisdom of regulations, and the costs are not useful in isolation because they nearly uniformly exclude regulatory benefits.

If there is an argument to measure regulatory impact, then the answer to this problem is not to stop counting but rather to count better. None of this is to imply that all of the current methods used to count regulations are irredeemable. Questions like patterns of regulation over time, within a subject area, or across presidential administration are relevant to questions besides cumulative regulatory impact and its effects on businesses and individuals. The measured costs and benefits of regulations are also useful in this regard. And as discussed above, it is important to note that counts of OMB significant regulations and Unified Agenda entries are almost certainly superior to Federal Register documents, CFR pages, and the appearance of certain words in the CFR.

But none of these measures gets at the true question of how are businesses bear the costs of regulation and individuals who reap their benefits affected by regulations cumulatively. Are businesses that would otherwise be thriving, struggling under the burden of regulatory compliance? What investments and opportunities are firms, small and large, bypassing in order to devote resources to following the dictates of the state?

On the other side of the regulatory equation are individual regulatory beneficiaries. Which neighborhoods are cleaner as a result of EPA regulatory efforts? Do the people in those neighborhoods also enjoy safer workplaces and easier access to credit because of regulation? Or, as regulatory critics have contended, have they lost jobs because firms were forced to lay off workers in order to devote resources to compliance? Similarly, do they now have to pay higher prices for essential goods (Dudley and Mannix, 2018)? Disadvantaged individuals are also negatively impacted by regulations restricting access to benefits to which they would otherwise be entitled (Herd and Moynihan, 2019). Rarely, if ever, have scholars tried to measure the cumulative time individuals spend filling out government-required forms. Current regulatory counts and even most studies measuring cumulative impacts leave these questions unanswered.

Recent efforts by the Office of Management and Budget to amend Circular A4, the document that provides guidance for agencies producing Regulatory Impact Analyses, to include more detailed advice on the distributional impact of regulations (OMB, 2023a) will help in this regard. But, even if it is maintained by future administrations less committed to distributional analysis, it will be a considerable amount of time before these analyses improve to the extent that they can be aggregated in any meaningful way. And, of course, such analyses will be subject to the same limitations described above for current RIAs, independent agencies, and regulations that do not meet the threshold for requiring an RIA will produce no data.

In order to better understand the real impacts of regulations on the fortunes of businesses, and on the lives of individuals, the counts described above should be supplemented in two ways by scholars. One involves mining existing data—the massive amount of information contained in submissions to OMB under the Paperwork Reduction Act using new and sophisticated techniques. The other involves research which would instead emphasize a much more microscopic focus. This could be quantitative studies of particular industries or communities or qualitative studies with deep dives into the experiences of individual firms or families. Below I briefly describe each approach in turn.

Learning from abroad and using the Paperwork Reduction Act and new technology

As noted above, the Paperwork Reduction Act is an underutilized resource for measuring the impact of regulation. Submissions by agencies under the PRA do not capture all regulatory requirements. However, the keeping of records and the filling out of forms are among the most salient requirements that the government imposes. And they are among the most annoying. It is through these requirements that many people have their most direct experience with the government (Shapiro and Borie-Holtz, 2020; Kaufmann and van Witteloostuijn, 2018). When someone fills out a form, they do not care if it is pursuant to a statute, a regulation, or agency discretion.

In addition, individuals applying for benefits to which they are legally entitled also experience the government through the forms they must fill out in order to secure such benefits. The agency submissions under the Paperwork Reduction Act will capture these burdens as well. In fact, because each agency is required to detail in a “supporting statement” in each PRA submission, (OPM, 2011) the rationale for its burden estimates, there exists a plethora of information regarding who bears these burdens and the associated benefits that that are intended to result from their imposition. Researchers could use this information to parse out cumulative impacts on both firms and on individuals.

The task would not be an easy one, theoretically requiring the examination of thousands of documents. These thousands of documents would be an excellent place to apply large language models and other scraping technologies. One could start with a particular industry or collection of individuals and focus on the information collections from the government that impact them. But even broader studies are possible as has been shown in other countries.

OECD countries have applied the SCM to isolate and estimate administrative burdens in order to reduce them. The Netherlands conducted a detailed, data-intensive survey of administrative burden to conclude that administrative costs stemming from regulation comprise 3.6% of the Dutch GDP (Cordes et al. 2022). The OECD extrapolated from the Dutch results and conducted simulations that suggested reducing the administrative burden by 25% would increase GDP in EU countries from between 1% and 1.7% (Tang and Verweij, 2004).

The SCM was developed by the OECD to measure the administrative costs associated with European Union regulations. If it, or an appropriate analog, were to be used in the United States, it would need to be adapted to the particularities of US regulation and to the data available through the Paperwork Reduction Act. Paperwork requirements would need to be sorted by who bears their burdens. Some of the estimates of burden may need to be verified. The use of the SCM by the EU primarily serves here as a proof of concept (OECD, 2014).

Using the SCM, large language models, and the PRA would be an improvement on current methods used to measure regulatory impact. But doing so would still be an incomplete measure. It would omit state and local regulations, which often overlap with federal regulations and require duplication of effort. This duplication is often a source of hostility toward the government and is almost never studied in the literature (Shapiro and Borie-Holtz, 2020). And most importantly, it would likely do nothing to account for regulatory benefits. To meaningfully do this requires a deeper examination focused on a smaller subset of those impacted by regulation.

Learning from more focused studies and qualitative research

In the debates over the impact of regulation, too often, we have used aggregate numbers without understanding what they mean for those who benefit from regulation. In 2022, the Federal Register contained 80,756 pages 21,750 of which were devoted to rules. ¹⁹ This included 3168 rules, a 28.5% decrease from the previous year.²⁰ In 2021, the CFR contained 188,343 pages, a 1.24% increase over the previous year.²¹ These seem like big numbers. But (even aside from the concerns about their applicability and accuracy outlined above) what do they mean? How many of these regulations actually make a difference in hampering productivity or improving living conditions? How many improve public welfare?

Questions of the impact of regulation on employment and productivity have spurred some work in this area. In particular, there have been several useful studies about the effect of environmental regulations on the power sector (Coglianese et al. 2013). Interestingly, this work finds minimal regulatory impacts on employment. Work on immigration regulation in contrast has shown that regulation can have considerable negative impacts on the affected industries (Pham and Pham, 2010). More of this type of work is needed.

Even rarer has been qualitative work on the impact of regulations. In one study that included both a survey of small business owners and intensive interviews of a small number of owners, researchers found that, indeed, paperwork burdens were a challenge for small businesses and that many regulations imposing these burdens likely had little impact beyond embittering business owners against regulation generally. Overlapping regulations from different jurisdictions were also seen as a significant concern (Shapiro and Borie-Holtz, 2020). Kaufmann and van Witteloostuijn (2018) supplement their quantitative analysis by asking firms in the Dutch natural gas industry about their experience. These are the types of insights about cumulative impact that are absent from studies using the counts of regulation.

While a movement toward using A.I. and Large Language Models could doubtlessly be useful, we should note that the trend described above toward ever more detailed counts of words and phrases in regulatory text has only produced new errors and misunderstandings. We should take advantage of technology (for example, in analyzing the thousands of Paperwork Reduction Act submissions and supporting statements), but it is a complement, not a substitute for good qualitative work on regulatory impacts. “There is no doubt that these automated methods are no substitute for careful thought and close reading and require extensive and problem-specific validation,” (Franchino et al. 2024, p. 700).

Qualitative research is both time-consuming and can raise questions about generalizability. Following well established protocols in conducting qualitative research is essential (Rubin and Rubin, 2011). For these reasons, it can’t be the sole approach toward gaining an understanding of the collective impact of government regulation. But it also has the potential to provide insights that are missing from studies that rely on aggregate counts. How do regulations affect individual businesses and communities? Let us ask people in those businesses and communities. And to my knowledge, no such studies have been conducted on regulatory beneficiaries whether at the neighborhood or the family level. Similarly, I am unfamiliar with any studies that ask the question, how much time has this individual spent filling out paperwork to get government benefits (or to vote), and how does that affect their ability to improve their economic welfare.

Conclusion

The question of the impact of regulations is an important one. That importance has only grown as the reach of the regulatory state has grown, and regulation has become an increasingly important policy-making tool in an era of hyper-partisanship in Congress (Curry and Lee, 2020). However, hard-to-answer questions about regulatory impact lend themselves to answers that are overly simple and easy to demagogue. As tools for text analysis advance, it will be easier and easier to come up with numbers representing the reach of the regulatory state. Just because these answers are easier to come up with, doesn’t mean they are any more accurate than the overly simplistic measures of the past.

Simplistic and arguably incorrect answers about regulatory impact don’t indicate that the questions they are attempting to address are invalid. These questions have tended to be asked, however, in a manner that renders them incomplete. The cumulative impact of regulations on businesses, especially small businesses, is a valid concern that could affect business viability, prices, and employment (Mandel and Carew, 2013). It can harm organizational flexibility (Kaufmann and van Witteloostuijn, 2018). But while recent work has brought to light the issues of cumulative burden on disadvantaged populations that find it difficult or impossible to access benefits to which they are entitled (Herd and Moynihan, 2019), empirical examination of this burden is limited. And work on the cumulative benefits of regulation is non-existent.

The time is ripe for a more serious examination of all of these questions. Attention to the distribution of regulatory impacts has never been greater (Cecot, 2023). And this trend is likely to continue in light of the changes to OMB Circular A4 described above. However, attention to the distributional impacts of individual regulations will paint an unavoidably incomplete picture. Cumulative impacts of regulation have no natural home within the federal government (Coglianese et al. 2013). There is every reason to expect that cumulative distributional impacts will be treated similarly.

Continuing to track significant regulations reviewed by OMB is worthwhile as a comparative tool. Using the Unified Agenda to get a broader count of regulatory actions and their various classifications is also a promising if imperfect and time-consuming approach to counting regulations. And tracking the reported costs and benefits of regulations for which agencies actually make those calculations will still be useful.

But to get a more thorough and thoughtful picture of the overall impact of the regulatory state, new approaches are needed by scholars and advocates. Utilization of the vast trove of data available in agency submissions under the Paperwork Reduction Act is an important step in improving these counts. And qualitative research has the potential, if done well and carefully, to provide more meaningful answers to the questions that motivate the more comprehensive studies. If we want to know how businesses and people are affected by regulations, maybe we need to ask business owners and individuals.