Reflections on Traceability for Visualization Research
Jen Rogers, Derya Akbaba, James Scott-Brown, Alexander Lex, Miriah Meyer
TLDR
This paper introduces "traceability" as a framework to ensure rigor and transparency in design-oriented visualization research, offering an alternative to traditional reproducibility.
Key contributions
- Introduces "traceability" as a framework for rigor and transparency in design-oriented visualization research.
- Investigates traceability through a collaborative autoethnographic reflection on years of research.
- Develops and tests tRRRacer, a tool operationalizing traceability's three tenets: Record, Report, Read.
- Provides a theorization of traceability and reflections on how to effectively support its implementation.
Why it matters
Traditional reproducibility fails for design-oriented visualization research. This paper introduces "traceability," a novel framework with a supporting tool (tRRRacer), to bring rigor and transparency to inherently unreproducible design processes. It provides a crucial method for documenting and assessing subjective, iterative work.
Original Abstract
Decades of advocacy for reproducibility and replication have advanced open, transparent practices in the sciences. However, traditional notions of reproducibility fit poorly with design-oriented visualization research, where insights emerge through subjective, situated, and iterative work. So how can we ensure rigor and transparency in processes that are inherently unreproducible? To introduce transparency in design-oriented research, we propose to focus on traceability: surfacing the origin and development of research contributions based on rich sets of artifacts documenting the design process. We investigated traceability through a collaborative autoethnographic reflection that builds on several years of work exploring ways to make design-oriented research transparent. This exploration includes an experiment to build a tool to support traceability, which we called tRRRacer. The tRRRacer tool provided a testbed for us to operationalize the three tenets of a traceable process: (1) Record abundant, annotated artifacts representative of research activities; (2) Report curated research threads that articulate rationale and evolution of the process, allowing others to (3) Read via interfaces that help retrace claims and assess plausibility. Reflecting on our experiences, we contribute a theorization of traceability and reflections on how we might support it.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.