What kind of instrument it is, and who holds it
A setback is a zoning creature. It comes out of an ordinance a county, city or town chose to adopt, under a statute in each state that says a locality may adopt one, and it lives with the zoning administrator or the planner rather than with the building official. Two separate questions, and a job can be square with one and crosswise with the other.
Because it is optional and local, there is no Virginia setback and no North Carolina setback. Both state statutes hand a locality the power to fix dimensions and neither prints a dimension itself. A number quoted to you as the state's is a number quoted from nowhere, and several counties around here have no county-wide zoning ordinance to hold one.
It is measured from a line, and the line is not always the one you expect
The ordinances read for this page measure from the lot line or from the edge of the street right of way. Surry County, North Carolina adds a third possibility that catches people out. Its setback runs to whichever is nearer of the lot line, the street right of way line, or the line where NCDOT's street maintenance begins. On a road out in the county those three can sit in three different places, and the closest one governs.
Then there is what the distance is measured to. Pulaski County, Virginia takes it to the vertical wall of a building or structure. A flat slab has no vertical wall for a tape to catch. The Town of Hillsville runs its setback line against the principal building. Different ordinances, different reference points, and both of those are on file and readable.
Whether it even reaches a slab is a definitions question
Since the distance is measured to a structure, the whole homeowner question turns on how that one word was defined in that one ordinance, and the definitions in this corner of the two states disagree with each other.
Surry County, North Carolina carries an exclusion list inside its definition, and it names at-grade slab patios and driveways among the things a structure is not, so the setback measured to a structure does not reach them. That is the setback only. The same ordinance keeps separate driveway standards of its own, and they are untouched by it.
Giles County, Virginia writes no exclusion list at all, in the copy of its ordinance posted on the county's own site: that definition reaches anything constructed or erected whose use requires a location on the ground, and a poured slab is constructed and does require one. The copy is captioned 1992 as amended and was posted in 2019, so confirm with the county before you build anything on it. Two counties, two different definitions, and whether the Giles one reaches a patio is a question for its zoning administrator.
So ask three things of the planning office and ask them in this order. Is this lot zoned at all. Which district is it in. Does your definition of structure reach a slab at grade. That third question is the one almost nobody asks and the one that decides it, on a patio or on a driveway alike. Nothing on this page is a figure, because no dimensional table for any jurisdiction here was read for it.
Why your neighbor's slab proves nothing
Concrete poured tight to a line down the road tells you nothing about your ground. It may be in another district, or inside a town, or over a county line where nobody zones anything. It may predate the ordinance in force today. It may sit on a variance granted on facts belonging to that lot. Or nobody ever asked anyone.
Keep the setback separate from the two things it gets confused with, and do not work either of them out from here. A recorded easement is a private right out of a deed, and the easement entry has it. The strip along a state maintained road answers to the state transportation department and takes a permission of its own. All this page will say about them is that clearing the setback clears neither. Our Carroll County page shows what one local desk actually puts in writing.