If you've ever needed a list of companies with their industry, size, headquarters and follower count, you've probably looked at LinkedIn and thought, “it's all right there on the page.” It is. Getting it reliably is another story.
This guide walks through a small, honest LinkedIn company scraper in Go using only the standard library and one HTML parser. It covers what works, what breaks, and where it makes sense to stop building it yourself. We'll only touch public company pages (the ones you can open in a private window without logging in). Scraping logged-in data, or profiles of private people, is a different legal and ethical conversation, and this article doesn't go there.
What a public company page gives you
Open https://www.linkedin.com/company/shopify in a private window. Without logging in, you usually get:
- Name, tagline and description
- Industry, company size, headquarters, founding year, company type
- Website and specialties
- Follower count
- Logo and cover image
Most of the structured bits live in two places in the HTML: a <script type="application/ld+json"> block and a handful of dt/dd pairs in the “About” section. JSON-LD is the friendlier of the two, so start there.
Project setup
mkdir li-company && cd li-company
go mod init example.com/li-company
go get golang.org/x/net/html
Step 1: fetch the page like a browser would
LinkedIn returns very different responses depending on headers. A bare Go http.Get almost always gets a redirect to the login wall or a 999 status code (LinkedIn's own “request denied” code). Send realistic headers and handle the status explicitly.
package main
import (
"errors"
"fmt"
"io"
"net/http"
"time"
)
var client = &http.Client{
Timeout: 20 * time.Second,
// Don't silently follow redirects to the auth wall.
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
var ErrBlocked = errors.New("blocked or redirected to login")
func fetch(url string) ([]byte, error) {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return nil, err
}
req.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0 Safari/537.36")
req.Header.Set("Accept", "text/html,application/xhtml+xml")
req.Header.Set("Accept-Language", "en-US,en;q=0.9")
resp, err := client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
switch {
case resp.StatusCode == 999, resp.StatusCode == 429:
return nil, fmt.Errorf("%w: status %d", ErrBlocked, resp.StatusCode)
case resp.StatusCode >= 300 && resp.StatusCode < 400:
return nil, fmt.Errorf("%w: redirect to %s", ErrBlocked, resp.Header.Get("Location"))
case resp.StatusCode != 200:
return nil, fmt.Errorf("unexpected status %d", resp.StatusCode)
}
return io.ReadAll(io.LimitReader(resp.Body, 5<<20))
}
Two details matter here. Disabling redirects means you see the authwall redirect instead of happily parsing a login page as if it were Shopify. And treating 999 as its own error lets your retry logic back off instead of hammering.
Step 2: pull out the JSON-LD
import (
"bytes"
"encoding/json"
"strings"
"golang.org/x/net/html"
)
type Company struct {
Name string `json:"name"`
Description string `json:"description"`
URL string `json:"url"`
SameAs string `json:"sameAs"`
Logo any `json:"logo"`
Address struct {
Locality string `json:"addressLocality"`
Region string `json:"addressRegion"`
Country string `json:"addressCountry"`
} `json:"address"`
Employees struct {
Value int `json:"value"`
} `json:"numberOfEmployees"`
// Filled from the About section, not JSON-LD.
Industry string `json:"industry,omitempty"`
Size string `json:"size,omitempty"`
Founded string `json:"founded,omitempty"`
}
func parseJSONLD(body []byte) (*Company, error) {
doc, err := html.Parse(bytes.NewReader(body))
if err != nil {
return nil, err
}
var found *Company
var walk func(*html.Node)
walk = func(n *html.Node) {
if found != nil {
return
}
if n.Type == html.ElementNode && n.Data == "script" {
for _, a := range n.Attr {
if a.Key == "type" && a.Val == "application/ld+json" && n.FirstChild != nil {
found = decodeOrg(n.FirstChild.Data)
}
}
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
walk(c)
}
}
walk(doc)
if found == nil {
return nil, errors.New("no Organization JSON-LD found")
}
return found, nil
}
// LinkedIn sometimes wraps the Organization in an @graph array.
func decodeOrg(raw string) *Company {
raw = strings.TrimSpace(raw)
var graph struct {
Graph []json.RawMessage `json:"@graph"`
}
if json.Unmarshal([]byte(raw), &graph) == nil && len(graph.Graph) > 0 {
for _, item := range graph.Graph {
var t struct{ Type string `json:"@type"` }
json.Unmarshal(item, &t)
if t.Type == "Organization" {
var c Company
if json.Unmarshal(item, &c) == nil {
return &c
}
}
}
return nil
}
var c Company
if json.Unmarshal([]byte(raw), &c) == nil && c.Name != "" {
return &c
}
return nil
}
Note Logo any: sometimes it's a string, sometimes an ImageObject. Typing it strictly is the fastest way to turn a working scraper into one that fails on 10% of companies.
Step 3: the About section fallback
Industry, size and founding year often aren't in JSON-LD. They sit in definition lists on the page, labelled in English for English-locale requests:
func parseAbout(body []byte, c *Company) {
doc, err := html.Parse(bytes.NewReader(body))
if err != nil {
return
}
var lastLabel string
var walk func(*html.Node)
walk = func(n *html.Node) {
if n.Type == html.ElementNode && (n.Data == "dt" || n.Data == "dd") {
text := strings.TrimSpace(textOf(n))
if n.Data == "dt" {
lastLabel = strings.ToLower(text)
} else {
switch {
case strings.HasPrefix(lastLabel, "industry"):
c.Industry = text
case strings.HasPrefix(lastLabel, "company size"):
c.Size = text
case strings.HasPrefix(lastLabel, "founded"):
c.Founded = text
}
}
}
for ch := n.FirstChild; ch != nil; ch = ch.NextSibling {
walk(ch)
}
}
walk(doc)
}
func textOf(n *html.Node) string {
if n.Type == html.TextNode {
return n.Data
}
var sb strings.Builder
for c := n.FirstChild; c != nil; c = c.NextSibling {
sb.WriteString(textOf(c))
}
return strings.Join(strings.Fields(sb.String()), " ")
}
This is the brittle part. Labels change with locale, and the markup shifts a few times a year. Write a test against a saved HTML fixture so you notice when it breaks.
Step 4: a polite crawl loop
func main() {
urls := []string{
"https://www.linkedin.com/company/shopify",
"https://www.linkedin.com/company/stripe",
}
enc := json.NewEncoder(os.Stdout)
for i, u := range urls {
var c *Company
var err error
for attempt := 0; attempt < 3; attempt++ {
var body []byte
body, err = fetch(u)
if err == nil {
if c, err = parseJSONLD(body); err == nil {
parseAbout(body, c)
break
}
}
wait := time.Duration(5*(attempt+1)) * time.Second
fmt.Fprintf(os.Stderr, "%s: %v (retry in %s)\n", u, err, wait)
time.Sleep(wait)
}
if err != nil {
fmt.Fprintf(os.Stderr, "giving up on %s\n", u)
continue
}
enc.Encode(c)
if i < len(urls)-1 {
time.Sleep(time.Duration(3+rand.Intn(4)) * time.Second)
}
}
}
(Add math/rand and os to your imports.) Output is JSON Lines, which you can pipe into jq, SQLite or a spreadsheet.
The walls you'll hit
The 999 and the authwall. From a residential IP at low volume, public company pages mostly load. From a cloud server (AWS, GCP, Hetzner) the first or second request often comes back 999 or redirects to /authwall. Rotating user agents doesn't fix this, because the decision is mostly based on IP reputation.
Regional hosts. Links like ca.linkedin.com/company/... or se.linkedin.com/... show up in search results. They resolve to the same company, but the page may come back in another language, so your dt labels won't match. Normalize to www.linkedin.com and send Accept-Language: en-US.
Partial pages. Sometimes you get a 200 with a stripped page: name and logo, no About section. Don't treat that as success. Check that the fields you care about are actually present, and retry later if they're missing.
Posts and employees. Company posts and the “employees” strip load through authenticated, internal endpoints. Getting them without logging in is unreliable, and doing it logged in breaks LinkedIn's User Agreement and puts your account at risk. If your use case needs posts, plan for that up front.
Legal. hiQ v. LinkedIn is often cited as “scraping public data is legal”. The actual outcome is narrower than that: hiQ ultimately lost on breach-of-contract grounds. Stick to public pages, don't log in with scraper accounts, don't collect personal data you have no reason to hold, and talk to a lawyer before you build a business on top of this.
When to stop building it yourself
The DIY version above is a good fit for a few hundred companies run from your laptop. It stops being fun when you need thousands of pages from a server, regional URLs handled for you, or company posts with pagination.
That's the point where a hosted API earns its keep. With ScrapeAtlas*, a company page is one GET request, using the same net/http client you already have:
req, _ := http.NewRequest("GET",
"https://api.scrapeatlas.com/v1/linkedin/company?url="+
url.QueryEscape("https://www.linkedin.com/company/shopify"), nil)
req.Header.Set("x-api-key", os.Getenv("SCRAPEATLAS_API_KEY"))
resp, err := http.DefaultClient.Do(req)
The response includes name, industry, size, founded, headquarters, followers, employeeCount, specialties and similarPages, plus a coverage object that tells you whether the result was complete. So the “partial page” problem from above becomes a field you can check. There's also /v1/linkedin/company/posts for paginated company posts. Each successful request costs one credit, and failed requests cost nothing.
Whichever route you pick, store raw responses, check completeness explicitly, and keep your request rate low. Those three habits save more time than any parser trick.
*I build it.