The Weekend Dialect Atlas: Mapping English Microdialects in the UK via Twitter in 48 Hours

Traditional dialect surveys take years and armies of fieldworkers. This project asks: can we get a roughly comparable result in a single weekend using geotagged Twitter data? The plan is to spend Saturday harvesting tweets containing known dialectal lexical items (e.g., 'cob', 'bap', 'butty' for bread; 'ginnel' vs 'alley' vs 'snicket' for passage) using the Twitter API, focusing on the UK. Then filter to only those with precise geocoordinates. On Saturday night and Sunday morning, we'll run a clustering algorithm (DBSCAN on geographic coordinates) to find hot spots of lexical usage, and for each cluster calculate a 'dialect signature' vector of normalized word frequencies. By Sunday afternoon, create an interactive map (using Leaflet/Mapbox) where each cluster is a colored polygon, and clicking reveals the top distinctive words. The output is not a peer-reviewed atlas but a 'what can we learn in 48 hours?' proof-of-concept. The diversity angle is literally the time constraint: the entire methodology is optimized for speed, embracing messiness and small sample sizes, which challenges the usual slow, meticulous norms of academic dialectology.

Existing dialect maps (e.g., the Survey of English Dialects) are decades old, costly to update, and miss urban and online variation. There is no real-time, community-driven map of how people actually speak today. Meanwhile, geotagged social media data is abundant but underused for microdialect mapping because researchers tend to plan multi-year grants. This project tests the hypothesis that a focused, weekend-long hack can produce useful preliminary maps, democratizing the research process.

• Collect at least 10,000 geotagged tweets containing predefined dialectal keywords across the UK in 24 hours.
• Clean and geolocate the dataset (filtering for UK, removing bots/manual noise).
• Run DBSCAN clustering to identify dialect hotspots with a minimum cluster size of 20 tweets.
• Generate for each cluster a 'top-10 distinctive words' list using TF-IDF compared to background corpus.
• Visualize clusters on a live, interactive map with popups showing the lexical signatures.
• Publish the code, data, and results as a GitHub repository by Sunday midnight.

• NOT a comprehensive or statistically robust dialect survey — sample bias and noise are accepted.
• NOT a permanent atlas — the map will decay as data ages.
• NOT covering non-English languages or global dialects (UK English only for the weekend).
• NOT human-reviewed validation of clusters (automation only).
• NOT a mobile app or polished UI — just a research artifact.

Data Collection: Python 3.9, Tweepy (Twitter API v2), geopy for reverse geocoding. Processing: Pandas, NumPy, scikit-learn (DBSCAN, TfidfVectorizer). Visualization: Folium (Leaflet via Python), or raw Leaflet.js + Flask. Storage: SQLite / GeoJSON files. Coding environment: Jupyter notebooks for prototyping, VSCode for final script.

The system is a pipeline with three stages:

1. Scraper (Saturday 8am–8pm): A Python script streams tweets containing a curated list of ~50 dialectal keywords (e.g., 'traybake', 'scran', 'butty', 'jitty'). It filters by place_country: 'GB' and includes geo.coordinates. Tweets are dumped into a SQLite table.
2. Filter & Cluster (Saturday 8pm–Sunday 2pm): Remove tweets without precise coordinates (lat/lon). Run DBSCAN with eps=0.02 (about 2km) and min_samples=20. For each cluster, extract all tweet texts and compute TF-IDF over the keyword list to get a cluster 'dialect vector'. Merge clusters that are adjacent and lexically similar.
3. Map Build (Sunday 2pm–8pm): Convert clusters to GeoJSON polygons (convex hulls). Serve a Leaflet map via a simple Flask app. Each polygon is colored by a hash of its top word; clicking shows the top 10 words with bar chart. Deploy to a free Heroku or Netlify static site. All code and data pushed to GitHub.

• Twitter API rate limits: only 500,000 tweets/month on free tier — may not get enough geotagged data in 24h. Mitigation: use multiple keyword queries, start early, and accept a smaller dataset.
• Geolocation sparsity: only ~1% of tweets are geotagged. Could end with <1000 usable tweets. Mitigation: also use place-names and reverse geocode, but that adds time.
• Bots and spam tweets distort clusters. Mitigation: filter out tweets with very common bot patterns (e.g., same text repeated, high frequency from one user).
• Clustering algorithm may produce meaningless groups due to noise. Mitigation: manual sanity check on Sunday afternoon, possibly merge or discard clusters below lexical coherence score.

• Is 48 hours enough time to clean data meaningfully? Or will the clusters be too noisy?
• How do we define dialectal keywords? Rely on existing literature or crowdsource? A quick crowd-list may be more inclusive but less rigorous.
• Should we weight tweets by user to avoid one prolific user dominating a cluster? (Probably yes, but adds complexity.)
• Could we use bounding-box sampling instead of keywords to capture all tweets in an area, then derive dialectal words via unsupervised topic modelling? That would be more ambitious but likely too slow for a weekend.

The weekend arrived and real life intervened — a friend's wedding, a burst pipe, and a looming work deadline. The plan remains a perfect what-if: a compact, exciting idea that could have shown how fast language changes propagate in social media. Maybe next bank holiday.

The diversity angle (weekend build) is both a gimmick and a serious constraint that forces methodological creativity. The project is essentially a hackathon for dialectology. If I were to actually do it, I would pre-curate the keyword list the week before and have the scraping code ready to go so Saturday morning is just hitting 'run'. Also, I'd avoid dealing with real-time API challenges by using a pre-collected corpus if available, but that would violate the 'single weekend' purity.