When exploring new technologies for local governments, it's important to have trusted resources. For AI specifically, enter the Gov AI Coalition.
The GovAI Coalition is composed of government members from local, state, and federal agencies united in the mission to promote responsible and purposeful AI in the public sector. As agencies look to AI to improve services, the GovAI Coalition strives towards a more just and beneficial technological future for all. AI tools have the potential to make government services more responsive, efficient, and accessible to all, but AI also poses serious risks.
The Coalition keeps a vendor registry with important information on security, data management, other technology specifics to help them vet solutions. Here is the completed vendor fact sheet for Polimorphic. You can also view a PDF here.
Vendor Name: Polimorphic
System Name: Polimorphic AI
Overview: Polimorphic AI is an AI-powered voice, search, and chat interface designed specifically for local governments. It enables constituents to interact with their local government by asking questions and receiving accurate, timely responses. The platform leverages advanced language models and Retrieval Augmented Generation (RAG) techniques to query and provide information from municipal data sources such as official websites, PDFs, and fact sheets. The Voice AI system, in particular, leverages advanced language detection and translation capabilities, allowing constituents to communicate in multiple languages and enhancing accessibility for non-native speakers.
Purpose: Polimorphic AI performs the following functions:
- Voice Interaction: Allows constituents to use voice commands to ask questions and receive spoken responses, supporting multiple languages and dialects.
- Call Triaging: Voice AI uses fuzzy logic to direct callers to the correct department, including as a fail-condition if AI-based query response is insufficient/not found within training data. This replaces phone trees.
- Search Interface: Enables users to search for information using text-based queries.
- Chat Interface: Provides a conversational chat experience for users to interact with the AI system.
- Process Automation: Assists users in processing applications and form filling, simplifying bureaucratic procedures.
- Accessibility: Enhances accessibility by making government services available through voice interaction, benefiting individuals with disabilities or those who prefer audio communication.
- Purpose: The platform aims to enhance constituent engagement by providing easy access to local government information, improving transparency, streamlining communication between citizens and municipal authorities, and making government services more accessible to all constituents.
Intended Domain: The AI system is intended for use in the local government information services domain, facilitating communication and information dissemination between municipal governments and their constituents.
Training Data: The system utilizes the pre-trained GPT-4o model provided by OpenAI. We do not train the language model ourselves.
Data Used: The AI employs Retrieval Augmented Generation (RAG) to query data collected from municipal websites, official documents, PDFs, and fact sheets provided by the municipality.
Data Updates: Data is updated regularly as new information becomes available from the municipality. We have built tools to automatically update municipal data, ensuring that the AI system always provides the most current information.
Legal Compliance: All data used is legally obtained, and its use is fully licensed and compliant with local regulations.
Future Enhancements: In the future, we plan to provide access to training a fine-tuned model, allowing for more customized and precise interactions based on specific municipal needs.
Test Data: A set of frequently asked questions and scenarios based on constituent interactions are used to test the system - utilizing a combination of synthetic and real questions submitted by residents from multiple municipalities ranging from population counts of 1000, to over 1 Million.
Testing Conditions: The system was tested under various conditions, including voice queries, text searches, and chat conversations, to ensure robust performance across all interaction modes.
Validation: Responses were validated against verified municipal data to assess accuracy and reliability.
Model Information: The system uses the GPT-4o large language model provided by OpenAI, which is a transformer-based deep learning model.
Additional Techniques: Retrieval Augmented Generation (RAG) is employed to integrate specific municipal data into the AI's responses.
Built Upon: The platform is built on existing, state-of-the-art AI models to ensure high-quality interactions. Future iterations plan to include a “bring your own model”/multi-modal based on client preference.
Update Procedure:
Model Updates: Since we use the GPT-4o model provided by OpenAI, updates depend on OpenAI's release schedule. Updates are automatically integrated into our system.
Data Updates: Municipal data sources are updated regularly (based on client preference) to reflect the most current information.
User Choice: Users automatically benefit from model improvements; there is no option to revert to previous versions.
Documentation: Detailed documentation on new features and updates is available on our website and through our support channels.
Inputs and Outputs:
Inputs:
- User queries via voice commands, text search, or chat messages.
Outputs:
- AI-generated responses providing relevant information based on municipal data.
- Responses can be delivered in text or synthesized speech.
Interfaces and Integrations:
- Web-based interfaces.
- Integration with existing municipal platforms and services.
- Supports API integrations for third-party services.
Performance Metrics:
Performance Metrics:
- Response Accuracy: Percentage of correct responses based on municipal data.
- Response Time: Average time taken to generate a response.
- User Satisfaction: Ratings collected through feedback mechanisms.
- System Uptime: Availability percentage of the platform.
Current Performance:
- Response Accuracy: 90% accuracy based on internal testing.
- Response Time: Average of 1.5 seconds per query.
- User Satisfaction: 4.5 out of 5 stars in user feedback.
- System Uptime: 99.9% over the past year.
Monitoring:
- Users and administrators can monitor performance via real-time dashboards and analytics tools provided within the platform.
Bias:
Potential Biases:
- Bias inherent in the pre-trained GPT-4o model.
- Biases in municipal data due to outdated or unrepresentative information.
Mitigation Strategies:
- Data Verification: Using official, vetted municipal data sources.
- Content Filtering: Implementing filters to detect and remove inappropriate or biased language.
- Regular Audits: Conducting periodic reviews of AI outputs to identify and address biases.
- User Feedback: Allowing users to report biased or inappropriate responses for prompt action.
Robustness:
Handling Outliers:
- The AI system prompts users for clarification when queries are ambiguous or outside the scope of available data.
- Employs fallback mechanisms to provide general information or direct users to human assistance when necessary.
Feedback Loop:
- Overwritten decisions and user feedback are collected and analyzed.
- Adjustments are made to retrieval mechanisms and data sources based on this feedback.
- System improvements are continuously implemented.
Optimal Conditions:
- When queries relate to information present in the municipal data sources.
- Clear and specific user questions.
- Up-to-date and comprehensive data availability.
Minimum Requirements: No strict minimum data quantity, but the relevance and quality of data directly impact performance.
Poor Conditions:
Queries about information not present in the data sources.Ambiguous, vague, or overly broad questions.Outdated or incorrect data in the sources.
Limitations:
- May generate responses that are not entirely accurate or may "hallucinate" information.
- Difficulty handling highly technical or legally nuanced queries.
Error Conditions:
- Errors are more likely when the AI must infer information not explicitly available.
- Complex queries that require deep domain expertise.
Mitigation Strategies:
We have implemented measures to guide the model to avoid excessive inference, thereby mitigating hallucinations. The AI is designed to provide responses based strictly on available data and to acknowledge when information is not available.
Explanations:
- Responses include references to source documents when appropriate.
- Designed to be clear and understandable to the general public.
Transparency:
- Municipal administrators have access to detailed logs and analytics.
- Users can request additional information or clarification within the interface.
Jurisdiction-specific Considerations:
Compliance:
- Adheres to all relevant local and state regulations regarding data privacy and accessibility.
- Ensures data handling complies with laws such as GDPR and CCPA where applicable.
Data Security:
- Implements robust security measures to protect sensitive information.
- Regular security audits and assessments are conducted.
How is the AI tool monitored to identify any problems in usage? Can outputs (recommendations, predictions, etc.) be overwritten by a human, and do overwritten outputs help calibrate the system in the future?
Monitoring:
- Automated logging of system performance and user interactions.
- Real-time alerts for unusual activity or performance degradation.
- User feedback mechanisms within the interface.
Human Oversight:
- Municipal administrators can review all AI-generated content and "course-correct" using FAQs and prompt engineering.
- Continuous improvement processes are in place based on this feedback.
How is bias managed effectively?
- Regular audits of AI responses to detect and address biases.
- Content moderation filters to prevent dissemination of biased or inappropriate information.
- Feedback loops and human-in-the-loop processes for Voice AI.
- Data analysis and dashboards for staff to audit.
Have the vendors or an independent party conducted a study on the bias, accuracy, or disparate impact of the system? If yes, can the Agency review the study? Include methodology and results.
Studies Conducted:
While we have not had vendors produce formal, shareable studies, our customers have conducted their own quality testing and assessments. As a result, our product is now trusted and live with multiple municipal clients.
Availability:
- We can share aggregated results and insights from our clients' assessments upon request, respecting confidentiality agreements.
- Willing to collaborate with independent third parties for comprehensive studies.
Methodology and Results:
Detailed methodologies and results can be provided, outlining testing procedures, data used, and conclusions drawn.
How can the Agency and its partners flag issues related to bias, discrimination, or poor performance of the AI system?
Reporting Mechanisms:
- In-app reporting tools for immediate flagging of issues.
- Dedicated support channels via email and phone for agencies and partners.
- Regular meetings and reports to discuss system performance and address concerns.
Resolution Process:
- Prompt investigation of reported issues.
- Implementation of corrective actions and communication of resolutions to stakeholders.
How has the Human-Computer Interaction aspect of the AI tool been made accessible, such as to people with disabilities?
Accessibility Features:
- Compliance with Web Content Accessibility Guidelines (WCAG) 2.1 standards.
- Screen reader compatibility for visually impaired users.
- Keyboard navigation and voice command support.
- High-contrast modes and adjustable text sizes.
- Voice AI capabilities enable previously text based processes and forms to become more accessible, allowing users to interact with and complete forms through voice commands.
Assessments:
- Conducted usability testing with diverse user groups, including individuals with disabilities.
- Results indicated high levels of accessibility and ease of use.
- Continuous improvements are made based on user feedback to enhance accessibility features.
Please share any relevant information, links, or resources regarding your organization’s responsible AI strategy.
Responsible AI Strategy:
- Committed to transparency, fairness, and accountability in AI deployment.
- Policies and guidelines are available here.
- Regular updates and resources are provided to keep stakeholders informed.