Scrapbook: Screenshot-Based Bookmarks for Effective Digital Resource Curation Across Applications

poster version from 2020

“complicated as one needs to retrieve multiple resources across different applications” ⤴️
“sometimes, users need to retrieve digital re- sources across multiple applications to resume a computing task. However, the current methods - bookmarks, ﬁle systems, and recent history - for curating such resources are limited in their capacity and dispersed over multiple applications.” ⤴️
“Various applications currently provide “recently opened” functions for users to help them reopen and recon- struct an earlier working context. Nevertheless, this feature only works for a limited amount of time and for recent N docu- ments.” ⤴️
“a signiﬁcant amount of work is needed to organize ﬁles and web pages at scale” ⤴️
“Scrapbook, a software that allows users to curate digital resources with screenshots. Scrapbook allows users to take a screenshot of their computer screen and stores metadata of captured applications in the screenshot.” ⤴️
“users may not recall how to navigate the folder structure to ﬁnd the resources that they want” ⤴️
“Later, users can utilize multimodal (visual and textual) infor- mation to retrieve information that they want to recall or to restore the working context of a certain task.” ⤴️
“utilizing vi- sual history for mental reconstruction[1, 3, 6],” ⤴️
“Users can take screenshots of their computer to book- mark any digital resources that are apparent on-screen or to save all the resources of a task. Later, users can open digital resources from the curated list of screenshots with custom information that they entered when making the capture.” ⤴️
“utilizes the human’s capability of episodic memory” ⤴️
“restoring a working context can be more” ⤴️
“Cross-application content curation.” ⤴️
“multimodal (visual and textual) information can facilitate mental reconstruction” ⤴️
“we implemented an algorithm to detect applications not only the active (front-most) ones but also the ones that are partially visible while eliminating invisible software.” ⤴️
“When taking a screenshot, a user can enter textual messages, titles, and memos, to assist them in recalling” ⤴️
“With additional metadata - such as titles and memos, users can quickly recall more details about each captured screenshots.” ⤴️
“Collection View and Detail View” ⤴️
“reviewing screenshots of computer screen activities with corresponding metadata helps people recall past tasks quickly and can enhance users’ working productivity [14, 15, 9]. According to these ﬁndings, it is shown that computer users can quickly resume their previous working context with the combination of visual and textual cues[9, 6, 12].” ⤴️

paper version from 2022

“often use multiple resources to do a single task [2, 20].” ⤴️
“conventional curation methods like bookmarks, recent document histories, and file systems place limitations on effective retrieval. Such features typically work only for resources of one type within one application, ignoring the interdependency between resources needed for a single task.” ⤴️
“no single method that works for all types of resources.” ⤴️
“Scrapbook, a novel application for digital resource curation across applications that uses screenshot-based bookmarks. Scrapbook extracts and stores all the metadata (URL, file location, and application name) of windows visible in a captured screenshot to facilitate restoring them later.” ⤴️
“multimodal—visual and textual—data helped them recall past computer activities and reconstruct working contexts efficiently.” ⤴️
“o manage and restore resources of multiple types in a disjointed process” ⤴️
“identifying resources from largely text-based handles can be cognitively challenging. For example, a URL, a file name, or a web page title may not provide enough information for a user to recall the nature of the page’s contents. It takes workers extra effort to locate and label each resource carefully (e.g., by renaming bookmarks and files or organizing folder hierarchies) to facilitate later recognition and navigation.” ⤴️
“difficult to bundle heterogeneous resources from different applications into a single bookmark” ⤴️
“leaving windows open to eliminate retrieval tasks is a common solution, but doing this results in a cluttered computing environment and makes it more challenging to locate necessary resources among many open windows [24, 30].” ⤴️
“screenshots are displayed in Masonry layout [13] to roughly order them in a temporal order” ⤴️
“a title, screenshot, description, and date so that users can have multimodal cues for recognizing the digital resources contained.” ⤴️
“users can enter keywords in the search field at the top to find specific captures.” ⤴️
“o curate cross-application resources.” ⤴️
“Scrapbook offers a search feature that allows users to search their screenshots with metadata, which may contain information like application names and URLs.” ⤴️
“We use a dynamic bit-masking al- gorithm to identify all the visible windows in a screenshot, thereby also capturing windows behind the frontmost application.” ⤴️
“The participants also found the Collection View helpful as a glanceable to-do list that helps remind them of their tasks.” ⤴️
“s can later browse their collection of screenshots just as they might browse a photo album. A” ⤴️
“Some researchers distributed the effort needed to curate resources by inserting microtasks when users are interrupted [43]. Inserting such microtasks was helpful for task resumption as well [49].” ⤴️
“micro-productivity [7, 28, 38]” ⤴️
“cross-device work support, assisting users in quickly switching between tasks and curating resources on the go.” ⤴️
“Given the inevitability of interruptions and frequent changes in working con- texts, workers may encounter many resources of various types that they wish to curate, but curation activities—storing files, bookmarking web pages, opening files, and locating a web page—can degrade their working productivity.” ⤴️
”, researchers have found that as a user’s number of bookmarks increases, browsing, locating, and retriev- ing web pages becomes more challenging, ultimately requiring organizational efforts comparable to those applied to file systems: users must establish a hierarchical structure of bookmark folders, name bookmarks carefully, and categorize them immediately upon creation [1]. Cockburn et al. found that web users maintain large collections of bookmarks, nearly 200 on average, thereby yielding a lengthy list of bookmarks with folders [10]. The authors found empirical evidence demonstrating that users do not carefully curate bookmarks: imbalances between deletion and addition lead to the appearance of duplicate and invalid bookmarks. The limitations of native implementations of bookmarking may account for the popularity of commercial apps like Pocket 1 and Pinterest 2, which enable social bookmarking and provide multimodal cues (images and text) to facilitate web content curation. Similarly, collaborative web browsing and bookmarking was shown to be an effective cu- ration method for a specific task [27].” ⤴️
“t information scraps in personal information management systems should be easy to capture and discover [4].” ⤴️
“Digital curation activities can range from basic, low-level tasks like saving a file to complex, high-level tasks like creating and maintaining an organized hierarchy of nested folders.” ⤴️
“humans could correctly identify images of their own among those of others with over 80% accuracy” ⤴️
”. Scalable Fabric organizes thumbnail-like windows into groups near the edges of a computer’s screen for workers to quickly switch between windows by clicking a thumbnail of the window [45].” ⤴️
”, visual information can be processed in parallel, meaning that a human can recognize an image at a glance; conversely, humans cannot recognize a block of text visually and must parse it for meaning [47].” ⤴️
“Modern smartphone users take advantage of the ubiquity of smartphone cameras with the practice of taking photos to serve as external memory devices; for instance, a user might take a photo of a pillar to recall their car’s location in an airport parking lot or take a screenshot of mobile phone to save a QR code of a flight ticket [16].” ⤴️
“Taskposé arranges thumbnails of windows based on their relevance to an ongoing task, helping users to manage multiple windows based on usage patterns and transitions between them [5].” ⤴️
“WindowScape, which allows users to capture a group of windows on a screenshot to switch between digital work spaces [42]. However, these systems only allow transitions between already open windows; consequently, they do not address the challenge of long-term resource curation and retrieval” ⤴️
“t the resource management features web browsers provide—namely, bookmarks and a browsing history—are not the most frequently used methods for revisiting websites. For regularly visited websites, users tend to simply type their URLs in the address bar; for irregularly visited sites, users repeat keyword searches to find the sites again” ⤴️
“Rule et al. investigated the benefits of using a visual history for mental reconstruction and arrived at design implications by exploring various design options (animation vs. still image, thumbnail size) and their effects [32, 33]. ScreenTrack generates a time-lapse video by capturing screenshots periodically” ⤴️
but auto-methods over-save and feel invasive
“they differ from Scrapbook in as much as they visualize screenshots in the temporal dimension rather than using them for explicit curation” ⤴️
“This result is consistent with a previous work which found that history and bookmarking mechanisms provided by browsers can be ineffective [1, 48].” ⤴️
“d bookmarks are “cumbersome to keep organized (P52).” Participants complained about having to give bookmarks recognizable names and place them in the right folders.” ⤴️
“[P20] “I name files something that makes sense at the time while I’m in a rush, but then have trouble recalling where I put it or what that name means."" ⤴️
“Many participants (58/125) reported that the need to track where they left off when they resume a task is their biggest concern—to reconstruct their mental context. Thus, knowing the context of what needs to be retrieved is the biggest challenge.” ⤴️
“f locating a desired bookmark within a large field of irrelevant ones” ⤴️
“they simply had too many bookmarks (20/125), and sometimes, they even forgot about what pages they had bookmarked (18/125)” ⤴️
“bookmarks are not scalable” ⤴️
“many participants (56/125) responded that they often leave windows open. In general, leaving too many applications open can impact a computer’s performance and create visual clutter,” ⤴️
“Users get lost in folder hierarchies.” ⤴️
“most commonly use their operating system’s native file manager (e.g., Finder, Windows Explorer) to curate files; 70 out of 125 chose Often or Always. Arranging files on the desktop was also identified as a common approach (Often/Always: 58 out of 125) to saving frequently accessed documents and files.” ⤴️
“we consider various actions inherent to curating digital resources— recording, browsing, searching, and retrieving—and we explain the design of Scrapbook as it relates to these actions.” ⤴️
“Scrapbook extracts all available metadata from each window visible in a screenshot.” ⤴️
but is it relevant metadata? is ‘as much as possible’ the right approach? maybe for speed ig
“metadata, such as file names, URLs, file paths, and application paths, is essential in reopening applications. We use an operating-system-level accessibility API to extract metadata. Combined with a screenshot, this metadata practically eliminates the challenge of locating resources immediately after recalling them from an image, which was one of three challenges we found in the formative study.” ⤴️
but that requires a lot of trust/permissions given to the app
”, Scrapbook will provide multimodal—visual, temporal, and textual—cues that users can rely on to locate a screenshot,” ⤴️
“Scrapbook identifies visible windows from a screenshot using a bit-masking algorithm.” ⤴️
“a screenshot as a handle to store digital re- sources in a bundle” ⤴️
”. In a more complicated case, a user may want to include a partially visible window in a screenshot, such a window header ofa relevant resource that is partially visible behind the main resource window” ⤴️
“it does not explicitly determine what portion of any window is visible.” ⤴️
“Capturing all open applications is another option, but this may force a user to remove/deselect irrelevant windows when users retrieve resources from a screenshot. Therefore, we identify the visibility of each window by running a bit-masking algorithm” ⤴️
“Scrapbook provides two screenshot capture options: capturing a screenshot by selecting an area (the first button in Figure 3) and taking a full-screen screenshot (the second button in Figure 3). The former is intended for use in targeting a specific application or a specific region ofa screen (e.g., a figure in an academic paper). These options resemble the capture options available in an operating system’s native screenshot functionality.” ⤴️
“dynamically maintain a matrix, denoted by A in Algorithm 1, that represents an area of interest considering the occlusion between windows.” ⤴️
”, examine the next window available until there is no window to examine or the area of interest is too small.” ⤴️
“there is a trade-off be- tween the screenshot size and the number of images that can be displayed within a full-screen window simultaneously.” ⤴️
“the Collection View, which shows a collection of screenshots; and the Detail View, which shows a single screenshot.” ⤴️
“timestamps presented as the time distance between the capture time and the current time (e.g., 3 days ago).” ⤴️
“marking important emails with a star, flag, or pin in an email client [18]” ⤴️
“Immediately after a user takes a screenshot, the user is given the opportunity to add their own input and select or dese- lect resources captured” ⤴️
“including in- visible resources can facilitate mental reconstruction.” ⤴️
“The Collection View uses a “Masonry” layout, pat- terned on online systems for curation like Pinterest, Google Keep, and photo albums [13].” ⤴️
“presented in re- verse chronological order” ⤴️
“We designed and implemented Scrapbook for Apple’s macOS.” ⤴️
“Scrapbook can curate any application and identify corresponding digital resources, provided that the application supports scripting.” ⤴️
“we designed the software architecture so as to facilitate remotely updating the AppleScript code by maintaining an external CSV file containing customized, application-specific scripts.” ⤴️
“All metadata and screenshots are stored in a local MongoDB database.” ⤴️
“We used Node.js, an open-source, back-end JavaScript runtime environment, to set up a local server for presenting user interfaces, screenshots, and metadata, and to provide a retrieval method in the browser interface.” ⤴️
“We developed Scrapbook as a window-less application that runs in a menu bar, allowing users to take a screenshot without having to locate the window (Figure 3).” ⤴️
“We collected metadata and screenshots only if participants agreed to share this data, as their screenshots could have included personal information.” ⤴️
“All partici- pants shared complete metadata; most ofthe participants shared the entire set of screenshots, while some shared a selection of screen- shots. Participants who declined to share some of their screenshots were concerned about the security ofpersonal information captured in the screenshots.” ⤴️
“64.2% of all captured screenshots were full- screen screenshots, and the rest were selected-area screenshots.” ⤴️
“[P8] “To keep a record for myself. A” ⤴️
“In one interesting use case, a participant captured a screenshot of their code editor as a way to capture a version of their code before revising it (Figure 6-Right). While the earlier version of the code would not have been available when opening the updated file from the screenshot, the screen captured one version that he believed it was stable. Interestingly, the user was actually able to select and copy code text from the image using their web browser’s built-in optical character recognition (OCR) functionality3, which fulfilled the purpose that the user intended.” ⤴️
“screenshots served more as short-term bookmarks that were useful to resume tasks or remind users oftasks to complete” ⤴️
”. Using a traditional bookmark, they bookmarked the front page and had to then navigate the course website in multiple steps if they wanted to resume a grading task:” ⤴️
”. In contrast, the screenshot bookmark they created in Scrapbook was a link to a specific student’s homework submission that they needed to resume grading from, resembling the original application of physical bookmarks: marking progress within a book, rather than merely marking the entire book.” ⤴️
“We realize that one of the critical values of Scrapbook is simply having a central repository for all screenshots.” ⤴️
“Making it easy to archive a screenshot, similar to ticking a checkbox in a to-do list, could be another way to support reminder-like usage of screenshot-based bookmarks.” ⤴️
“While it is not impossible to bookmark a specific page with a web browser’s bookmark functionality, it seems that the temporally organized collection view with visual cues encouraged participants to create transient bookmarks for task resumption, as opposed to bookmarking gateways to specific resources.” ⤴️
“making certain screenshots ephemeral could be useful to maintain a reasonable scale.” ⤴️
“While we explicitly mentioned that the system was a fully local application that does not communicate with other services, some participants (6/13) were concerned about personal information appearing in screenshots and wondered how much they could trust an app developer in practice.” ⤴️
“organization methods, such as filtering by time and type, searching, tags, and sorting, will be crucial in practice; manual curation (such as creating a folder hierarchy) should be avoided, as it is not scalable. Additionally, extracting semantic information from a screenshot and its metadata to automatically suggest a se- mantically meaningful title would minimize the effort expended on user-supplied information” ⤴️
strong disagree with these points
“One participant (P3) proposed that it would be excellent if Scrapbook could reproduce the window layout as well when restor- ing windows, which is feasible.” ⤴️
“an intelligent algorithm that can evaluate the relevance of inactive tabs to the visible tab based on proximity, how and when a new tab was opened, and transitions between tabs. Identifying interdependency between multiple tabs by usage pattern will be a promising research topic for reducing manual organization in tab bundling approaches [9, 19].” ⤴️
this isn’t enough - personally relevant connections aren’t always avaliable in this sort of metadata
“Screenshots can be used to support mobility by allowing users to distribute resources across devices, such as a home computer, computers at work, and tablets.” ⤴️
“screenshots can serve as a medium to share resources needed for both synchronous and asynchronous collaboration (e.g., sharing URLs and files as a bundle by sharing a screenshot).” ⤴️
“screenshots of heterogeneous types can coexist in the col- lection: some are temporary and others are permanent.” ⤴️
“recommend a screenshot that a user may want to curate.” ⤴️
no.