Delhi HC Cautions Departments on AI-Generated Judgments in Tax Proceedings

The Delhi High Court has cautioned the GST and Income Tax Departments about relying on Artificial Intelligence (AI) tools, such as ChatGPT, for legal citations in show cause notices (SCNs) and assessment proceedings. While the court did not annul the SCN under review, it raised concerns about the risks associated with using AI-generated judgments without proper human verification.

Risks of Relying on AI in Legal Proceedings

During the hearing, the division bench of Justice Prathibha M. Singh and Justice Shail Singh observed inaccuracies in the case references provided by the GST authorities. The court highlighted that AI-generated outputs could include:

• Misinterpreted legal principles
• Fabricated or non-existent case laws
• Conflicting or unreliable information
• Influences from AI’s training data or user prompts

Referencing its earlier ruling in Christian Louboutin SAS & Anr. vs Shoe Boutique, the court stressed that AI cannot replace human judgment in legal decision-making. It noted that AI tools may only assist in initial research or understanding, but they are not a substitute for verified legal authority.

Verification of Legal Citations is Essential

The bench instructed that all judgments included in SCNs and assessment orders must be independently checked before incorporation. It emphasized that departments are fully responsible for ensuring the accuracy of any AI-assisted references and warned against “AI hallucinations” that could mislead legal proceedings.

Expert Perspective

Tax and legal experts welcomed the judgment, noting that it clarifies AI’s role as a research aid rather than a source of authoritative citations. They stressed that unverified AI content cannot form the basis for legal presumptions or actions under GST or Income Tax law.

Conclusion

The Delhi High Court’s ruling underscores the importance of human oversight in legal processes. AI can support research, but ultimate responsibility and verification must remain with trained legal professionals to ensure accuracy, reliability, and fairness in adjudication.