Boolean Search String Experiment Follow Up

On November 8th, 2010, I wrote a post containing a Boolean search challenge and an experiment of sorts – I asked readers to share their approach and Boolean search strings for a basic job description. The inspiration for the experiment came from the fact that very few people seem to be consciously aware of the issue that when it comes to sourcing candidates via the Internet, resume databases, LinkedIn, etc., is that all Boolean candidate searches work, provided they are syntactically correct.

This is a fundamental problem which heavily influences the perception of sourcing as a low level, non-critical function and/or role, because anyone can take the title from a job description and the required skill terms, create a basic Boolean query, and get results. This leads to the idea that finding talent is easy – slap a few search terms together and voila! – you get candidates.

Congratulations for finding the same candidates everyone else is finding with the same unsophisticated searches. All candidate queries are definitely not created equal, and you simply cannot gain any competitive advantage running the same basic taken-straight-from-the-job-description title and keyword searches that everyone else does.

The lesser-known reality is that most people who run Boolean searches on LinkedIn, job board resume databases, in their Applicant Tracking Systems (if they even support Boolean – ouch!) and the Internet only find a small fraction of the talent that is available to be found. I’ve written quite a bit on the topic so I won’t belabor that point in this post.

The Boolean Search String Experiment Succeeded AND Failed

Unfortunately, in many organizations, sourcers and recruiters do not get (or seek out) the opportunity to compare and contrast their search strategies and tactics with their peers and/or managers on a position-by-position basis. Much of the magic of using Boolean queries for talent discovery and identification, or lack thereof, happens on each person’s computer screen and thus a great number of people have absolutely no public basis of comparison.

As someone who has trained several hundred recruiters and sourcers, I know that even for basic positions and hiring requirements, 10 different recruiters/sourcers will come up with 10 different search approaches, and while they will find some of the same people, each is likely to find some that none of the others will.

I wanted to show the world that point – to give people a public basis of comparison.

To date, the Boolean search string experiment has had 1,106 unique views with an average “time on page” of over 5 minutes, which is pretty solid for a short post. I was sincerely hoping I would get at least 100 responses to the Boolean search string experiment.

So far, only 36 intrepid people (thank you!) have shared their approach to analyzing the job description I used for the experiment and the Boolean search strings they would use to try and find qualified candidates – a whopping 3% of the number of people who have viewed the post. It’s also worth noting that 9 of the 36 work for the same company. :-)

As I and a few people have pointed out – there is no single best Boolean string or approach. However, I did see a number of  intelligently and logically constructed Boolean strings and well thought out analyses and search strategies.

Why the Low Response Rate?

I have several ideas as to why there were so few responses.

I’m only going to share one of them – I think most sourcers and recruiters aren’t truly confident in their Boolean search string skills.

I’ll put it this way – nearly everyone is comfortable dancing and singing when no one is looking and they are alone, but that all changes in a public forum.

However, if you know you’re a really good dancer or singer, most people have no problems dancing or singing in front of others. In fact – many enjoy performing because they take pride in their ability and they enjoy what they do.

The issue is that the only way someone can know they’re really good at  using Boolean queries for talent discovery and identification is if they have a basis of comparison. Sure – they may get good results, but how do they really know their skill level unless they have a chance to see what others would do if they were working the same positions?

If you were raised on a desert island in complete isolation and you taught yourself how to play the guitar – how would/could you know if you were actually any good at playing the guitar?

The reason why I won’t share any more of my theories as to why the experiment didn’t get more responses is that I’m actually more curious about your thoughts as to why there was such a low response rate. If you have any ideas, please share them.

My Analysis and Search Strategy

I was going to hold out for more responses, but I don’t think I would be accomplishing anything by doing such.

So, without further ado, here is the job description from the experiment:

Business Analyst

  • This mission critical role will involve you working with the inventory team to provide data analysis, reporting and technical expertise to meet business objectives. You will work directly with the inventory control group to provide the technical needs as driven by the business, and you will be required to provide business analysis support to the eCommerce and retail groups.

Required:

  • A minimum of 3 years of experience as a Business Analyst
  • Strong data analysis skills
  • Crystal Reports experience preferred
  • BSCS or related degree and/or experience
  • Experience with enterprise systems

My Analysis/Observations

While I don’t typically prefer to search by title, “Business Analyst” is a fairly common title used to describe this kind of work. However, I am aware that other companies may call people performing this role many things, including “business systems analyst,” “system analyst,” “systems analyst,” “requirements analyst,” “functional analyst,” “IT analyst,” or simply “analyst” or “consultant.”

You may be surprised to learn that my testing on this position specifically, 50% of the results mentioned “business analyst” and none of these (“systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”). The other 50% mentioned one of these (“systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”), and NOT “business analyst.”

The even 50/50 split threw me off, but I checked it 3 times. :-) That means that if you only searched for the title “business analyst,” you would be missing 50% of the available results, and you wouldn’t even know it.

From the job description, I can see this person will be primarily responsible for data analysis and reporting, and likely using Crystal Reports for the reporting, otherwise it wouldn’t likely be listed as preferred experience. Also of note is that this person will be working with the inventory control group as well as supporting the eCommerce and retail groups.

While experience with enterprise systems is listed as a requirement, I would not likely including it in my search efforts without speaking with the hiring manager/team for more clarification. There is no way to know if the reference to “enterprise systems” means something like experience with PeopleSoft, Oracle E-Business, SAP, etc., or something else. Outside of the major ERP systems, “enterprise systems” isn’t something many people will explicitly state in their resume or LinkedIn profile, and searching for it (or simply “enterprise”) can also result in many false positives. I’ll assess this qualification on the phone with the people I find.

(Some of) My Searches

I would choose to search my ATS and Monster first, given that I can take advantage of proximity search, the sheer volume of the sample size (millions to 10’s of millions – and there aren’t as many resumes on the Internet as you might think) and data/information depth, which is a weakness of LinkedIn.

For my first searches, I would not use any date range – I would search all resumes.

Search #1

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and inventory and retail and (ecommerce or “e-commerce” or b2b or b2c)

Search Notes:

I thought about searching for the common title denominator of “analyst,” but my experience and some testing has shown me that that results in many false positives (e.g., programmer analyst).

Even though the job description doesn’t require or even desire it, the ideal candidate would have some retail experience. However, this is very tricky to isolate, as most people won’t explicitly mention “retail,” and any retail experience they have will often represented in the companies they have worked for (e.g., Nordstrom’s, which has online retail, aka e-commerce, btw).

You can’t practically search for every retail company in existence, and using a “retail” industry filter on any site will only yield a small fraction of all people who actually do have retail industry experience.

Similar to the retail experience, even though it is not required or even preferred, I chose to search for (ecommerce or “e-commerce” or b2b or b2c), simply because if that’s the kind of group that this person will be responsible for supporting, why not try first to find people who have done it before? While those are some of the most common ways to express e-commerce experience, I could have added EDI, e-tail*, e-business, electronic funds tranfer, online transaction processing, credit card payments, etc., to the “or” statement.

I chose to search directly for the term “inventory” first, with the realization that some people may simply mention supply chain management/SCM as the only hint in their resume/profile that they may have experience with inventory systems. As such, I would plan on taking this approach if necessary after I exhaust all “inventory” searches. However, I also know that just because someone mentions supply chain management, it does not automatically mean they have inventory systems experience – there are plenty of other aspects/stages of SCM than just inventory.

I did not search for “Crystal Reports” because some people will only make reference to “Crystal Enterprise,” Crystal v.X, SAP Crystal, or Crystal Xcelsius. Thus I chose to work with the common denominator of “Crystal” at the expense of a few false positive hits of people with the name Crystal (who will, by the way, have everything else I am searching for). Crystal Reports is fairly common and widely used, however, if I didn’t find enough people with Crystal Reports, I would likely search for other popular reporting applications (“business objects,” “reporting services,” SSRS, Cognos, Hypersion Essbase, etc.)

This search only returns a very small number of results (which is exactly what I wanted – the bulls eye) – so once I have cherry-picked all of the people I want to contact that have obvious clues that they have everything my manager/client could possibly ask for (even though they didn’t explicitly ask for some – but I aim to exceed expectations by always searching for “maximum qualifications” first).

After I’ve gone through the first search’s results, I begin to systematically NOT out the non-required skills and experience, never seeing the same results twice, until my last search is limited to the minimum qualifications.

Search #2

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and inventory and not retail and (ecommerce or “e-commerce” or b2b or b2c)

Search #3

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and not inventory and retail and (ecommerce or “e-commerce” or b2b or b2c)

Search #4

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and inventory and retail and not (ecommerce or “e-commerce” or b2b or b2c)

Search #5

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and not inventory and not retail and (ecommerce or “e-commerce” or b2b or b2c)

Search #6

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and not inventory and retail and not (ecommerce or “e-commerce” or b2b or b2c)

Search #7

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and inventory and not retail and not (ecommerce or “e-commerce” or b2b or b2c)

Search #8

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and not inventory and not retail and not (ecommerce or “e-commerce” or b2b or b2c)

Search Progression

After searching sources that allow me to leverage proximity search and have a huge volume of deep data, I would then go to other resume databases (no proximity and the same or smaller volume of deep data), then LinkedIn (large volume of mostly shallow data), then the Internet.

However, I have to say I likely would not even have to leave my own database and go to a job board, LinkedIn, or the Internet because I would have found a decent quantity of very nice matches that I would begin messaging, contacting, assessing, matching and recruiting.

I also would not likely get past search #3 or #4 to find 20-30+ people to call. This is important because every person I would contact would have at least 2 out of the 3 implicitly desired skills (implied by the job description, but not explicitly stated in the requirements).

Final Thoughts

There is no single “best” search string for any given position, although one could argue that some search strategies and approaches are better than others, in terms of more quickly getting to and uncovering people are are highly likely to be a very strong match for a given role, taking required and desired (explicitly mentioned as well as implicit) skills and experience into consideration.

You should always wonder about whether or not your search strategy and specific queries are uncovering the strongest potential candidates that you have access to. The fact that every search “works” is a significant yet oft-overlooked issue, giving many the false perception that whomever is returned from the queries is all that is available to be found.

Perception is reality for most, but subjective reality isn’t objective reality – there are always more people to be found than are actually found by sourcers and recruiters. Always.

I will likely post another Boolean search string experiment in the future – please share your searches and encourage others to do the same.

Thanks!

  • Lois Grimshaw

    Excellent article Glen, thank you for taking the time to do this. I think we can all learn from each other

  • Pingback: Tweets that mention Boolean Search String Experiment Follow Up -- Topsy.com()

  • Thanks for the post Glen, insightful as always. My take on the low response rate is there is still a lack of enthusiasm and apathy when it comes to sourcing, and that you still have your fair share of people who believe that sourcing does not solve recruiting problems. A lot of people talk about talent acquisition, but you can’t acquire what you don’t identify and find in the first place. I know it’s an oversimplification.

    The more practical and logical conclusion behind the low response rate could be that there were several people who wanted to participate in the experiment, but saw all of the people who posted their comments and were intimidated to post their own, which brings to light your idea of the confidence/competence level in their Boolean search string skills.

  • A great case study…..

    Thanks Glen.
    Sarang

  • Hi Glen, Another great post. There’s so much to the art and science of recruiting. You’re a master scientist and I appreciate you sharing your expertise.

    In fact, this gets to reasons for the low response rate. I like your dancer/singer analogy. There are three other primary drivers in my view:

    1) Sourcers/Recruiters don’t want to give away their techniques, because they think this maintains a competitive advantage.

    2) Like recruiting itself, you are relying on a more passive approach of posting your ideas and expecting people to come to your site and respond with their ideas. Did you actively approach e.g. DM / email / call people to request they offer their ideas?

    3) It takes time/brainpower to think through and post. Not to mention the public scrutiny. For most the costs probably exceed the benefits.

    Again, appreciate your insights, sharing and expertise. I know I am continually learning from you and others in the recruiting space.

    Best, Mike

  • Arthur Fonzarelli

    IMHO I feel that there are not more responses due to the fact that there are a large number of recruiters who are not very skilled when it comes to Boolean Logic. I can tell u I know of at least 3 recruiters who sit within 12 feet of me that can barely run a “J2EE and WebSphere” search string.

    With all the training I have received over the years there has been very little emphasis on how a recruiter actually goes about sourcing candidates. Even on interverviews I have never been asked to run a search string. Questions always evolve around what types of candidates do you place, what tools have u used, how may placements do you make, etc.

    I have seen recruiter production drop significantly due to the fact that there are a number of client companies that are going the W2 route. Meaning u can no longer rely on just blasting sub-contractors like in years past…which means you actually have to search.

    There are people who poo poo the Boolean Logic as simply AND, OR, NOT but to me it is all semantics….getting creative.

  • Glen

    Thanks for the insights on complex Boolean Search strings. But, how about using Level #3 of Talent Mining process, as noted in your other blog piece…..directing the recruiters to the companies where the desired candidates are currently working. A haven for sourcing talent!

    I read your talent description and tried it in WANTED Analytics’ soon to be released, new talent sourcing web app. I basically used the search criteria, i.e. Business Analyst/Computer Systems Analyst, “Crystal Reports” in Retail Trade industry. Lets say, we do this for Chicago market. WANTED Analytics shows there are 171 such passive candidates in Chicago, skilled in Crystal Reports (using BLS data and modeling it with hiring demand data). Employers who had in the past posted job ads seeking similar talents are Walgreens, Rand Mcnally, Wickes Furniture, Sophus Tech., CDW, Abbott, Pinnacle Food Products, etc. We look at the historical job postings and they strike similar to what is being sought now. There could be people working at those companies in Chicago who the Recruiter may approach.

    A completely different tactic, but helps to land with appropriate talent once they can be located by companies, using LinkedIn or major job-boards’ resume data bank.

    Harold

  • William

    Great post and great site. I just recently discovered this site and I am really looking forward to learning from this community.

    I agree with many of the previous comments that the lack of participation in the study stems from ignorance, of not only the process, but also the potential of thoughtful and strategic sourcing. I work for a very large staffing organization and I know first hand that sourcing is often seen as nothing but a means to an end and there is little training or development around it. We have two new recruiters, one on their 6th week and one on their 13th week, and until I gave them a brief intro in building a search string last week, they didn’t even know they could use parentheses…

    I assumed that I wasn’t the only person that viewed constructing a search string as challenging and rewarding but hadn’t met any of those people yet.

    I’m looking forward to gaining expertise from Glen and everyone else.

  • Glen,

    Thank you so much for sharing. What a great holiday gift for all of us to open. The gift of learning. I appreciate the time you take to you put into these posts and your detailed analysis and expertise which help move the sourcing community forward.

    Best wishes for a successful 2011.

  • Jasonorton

    Sorry, missed the update. I’m impressed. AND I feel better about my own now. As you said, I’d be willing to bet that many are uncomfortable with their skill level and don’t want to, “dance in public”.

  • Santukamble

    What is the significance of  * asterik symbol

  • Pingback: Search strategies considered harmful | Information Interaction()