Abstract
This project was initiated by researchers at the TIK Centre for Technology, Innovation and Culture [https://www.sv.uio.no/tik/english/] with the aim of developing a platform to simulate survey responses using large language models. The platform is designed to explore how different populations might answer a standardized questionnaire. It reads real-world demographic records from an existing survey dataset, converts them into individualized prompts, combines these prompts with follow-up questions, and sends them to different LLMs through their respective APIs. The generated responses are linked back to the original demographic records and stored in structured formats for further analysis. By logging both inputs and outputs, the system supports transparency and repeatability in research workflows. The LLM-predicted responses are later validated against real survey responses, and their performance is compared with classical machine-learning-based prediction methods.
Background
Surveys are useful research tools, but they can be expensive and slow to run at scale, especially when comparing multiple countries, demographic groups, or questionnaire designs. This workflow provides a controlled way to study survey design, question wording, and response patterns by using language models as simulated respondents. In this setting, real respondent attributes are used to condition the model prompts, making it possible to explore how simulated response distributions vary across populations and experimental configurations. Further inquiries regarding the background of the project can be directed to Professor Fulvio Castellacci [https://www.sv.uio.no/tik/english/people/aca/fulvioca/].
Methodology
The project is organized to support transparency and reproducibility through configurable sampling, structured logging, and editable prompt templates. These templates separate the survey content from the code, making it easier to iterate on questions without changing the processing pipeline. A configuration file centralizes settings such as model choice, sampling rules, model parameters, and output paths, so different runs can be compared and repeated. Because the generated responses are stored in structured formats, the outputs can be analyzed with standard statistical tools. This makes it possible to compare simulated responses across countries or subgroups and to track how changes in prompts, models, or sampling rules affect the results. The approach can support exploratory research, pilot testing, and educational demonstrations of survey methodology.
As shown in Figure 1, the code first loads a dataset of respondent attributes and selects a subset of records based on user-defined sampling rules. Sampling can be restricted to specific countries, capped per country, and performed either randomly or sequentially, with a fixed random seed used for reproducibility. These choices help users control both the scale of the experiment and the comparability of results across runs. For each selected respondent, the system fills a context template with individual attributes, such as demographic or background variables. It then constructs a system prompt and appends a labeled list of survey questions from editable text files. This separation between respondent context and survey questions ensures consistency across respondents while keeping the questionnaire content easy to modify.
The prepared prompt is sent to the selected language-model provider using the model name and parameters defined in the configuration file. The model is instructed to return a structured JSON response, which is then parsed, validated, and saved. Each run produces a primary output file, along with timestamped logs of inputs and outputs, enabling auditing and later reprocessing. Alongside the responses, the workflow records the full prompt history for debugging and transparency. The logging structure makes it possible to trace each output back to the exact configuration and input record that produced it. This supports careful interpretation of results and helps identify potential sources of variation, such as differences in sampling, model settings, or prompt wording.
Overall, the pipeline is designed for repeatable experiments: prompts can be refined, models can be swapped, and sampling can be adjusted while maintaining a consistent data trail for analysis. Four main API-based models have been implemented in the workflow: OpenAI, Claude, Gemini, and Grok. Initial experiments have shown promising results regarding the ability of LLMs to simulate proportional response distributions. Figure 2 shows the proportional distribution of LLM-generated responses for a question with four response alternatives, compared with the corresponding real survey responses. This figure is intended as an example; a comprehensive analysis across multiple questions has been performed and the results will be reported separately in the future.
