HumanEval
The HumanEval benchmark is a dataset designed to evaluate an LLM’s code generation capabilities. The benchmark consists of 164 hand-crafted programming challenges comparable to simple software interview questions. For more information, visit the HumanEval GitHub page.
HumanEval assesses the functional correctness of generated code instead of merely measuring textual similarity to a reference solution.
Arguments
There are two optional arguments when using the HumanEval benchmark:
- [Optional]
tasks: a list of tasks (HumanEvalTaskenums), specifying which of the 164 programming tasks to evaluate in the language model. By default, this is set to all tasks. Detailed descriptions of theHumanEvalTaskenum can be found here. - [Optional]
n: the number of code generation samples for each task for model evaluation using the pass@k metric. This is set to 200 by default. A more detailed description of thepass@kmetric andnparameter can be found here.
By default, each task will be evaluated 200 times, as specified by n, the number of code generation samples. This means your LLM is being invoked 200 times on the same prompt by default.
Example
The code below evaluates a custom GPT-4 model (click here to learn how to use ANY custom LLM) and assesses its performance on HAS_CLOSE_ELEMENTS and SORT_NUMBERS tasks using 100 code generation samples.
from deepeval.benchmarks import HumanEval
from deepeval.benchmarks.tasks import HumanEvalTask
# Define benchmark with specific tasks and number of code generations
benchmark = HumanEval(
tasks=[HumanEvalTask.HAS_CLOSE_ELEMENTS, HumanEvalTask.SORT_NUMBERS],
n=100
)
# Replace 'gpt_4' with your own custom model
benchmark.evaluate(model=gpt_4, k=10)
print(benchmark.overall_score)
You must define a generate_samples method in your custom model to perform HumanEval evaluation. In addition, when calling evaluate, you must supply k, the number of top samples chosen for the pass@k metric.
# Define a custom GPT-4 model class
class GPT4Model(DeepEvalBaseLLM):
...
def generate_samples(
self, prompt: str, n: int, temperature: float
) -> Tuple[AIMessage, float]:
chat_model = self.load_model()
og_parameters = {"n": chat_model.n, "temp": chat_model.temperature}
chat_model.n = n
chat_model.temperature = temperature
generations = chat_model._generate([HumanMessage(prompt)]).generations
completions = [r.text for r in generations]
return completions
...
gpt_4 = GPT4Model()
The overall_score for this benchmark ranges from 0 to 1, where 1 signifies perfect performance and 0 indicates no correct answers. The model's score, based on the pass@k metric, is calculated by determining the proportion of code generations for which the model passes all the test cases (7.7 test cases average per problem) for at least k samples in relation to the total number of questions.
Pass@k Metric
The pass@k metric evaluates the functional correctness of generated code samples by focusing on whether at least one of the top k samples passes predefined unit tests. It calculates this probability by determining the complement of the probability that all k chosen samples are incorrect, using the formula:
where C represents combinations, n is the total number of samples, c is the number of correct samples, and k is the number of top samples chosen.
Using n helps ensure that the evaluation metric considers the full range of generated outputs, thereby reducing the risk of bias that can arise from only considering a small, possibly non-representative set of samples.
HumanEval Tasks
The HumanEvalTask enum classifies the diverse range of subject areas covered in the HumanEval benchmark.
from deepeval.benchmarks.tasks import HumanEvalTask
human_eval_tasks = [HumanEvalTask.HAS_CLOSE_ELEMENTS]
Below is the comprehensive list of all available tasks:
HAS_CLOSE_ELEMENTSSEPARATE_PAREN_GROUPSTRUNCATE_NUMBERBELOW_ZEROMEAN_ABSOLUTE_DEVIATIONINTERSPERSEPARSE_NESTED_PARENSFILTER_BY_SUBSTRINGSUM_PRODUCTROLLING_MAXMAKE_PALINDROMESTRING_XORLONGESTGREATEST_COMMON_DIVISORALL_PREFIXESSTRING_SEQUENCECOUNT_DISTINCT_CHARACTERSPARSE_MUSICHOW_MANY_TIMESSORT_NUMBERSFIND_CLOSEST_ELEMENTSRESCALE_TO_UNITFILTER_INTEGERSSTRLENLARGEST_DIVISORFACTORIZEREMOVE_DUPLICATESFLIP_CASECONCATENATEFILTER_BY_PREFIXGET_POSITIVEIS_PRIMEFIND_ZEROSORT_THIRDUNIQUEMAX_ELEMENTFIZZ_BUZZSORT_EVENDECODE_CYCLICPRIME_FIBTRIPLES_SUM_TO_ZEROCAR_RACE_COLLISIONINCR_LISTPAIRS_SUM_TO_ZEROCHANGE_BASETRIANGLE_AREAFIB4MEDIANIS_PALINDROMEMODPDECODE_SHIFTREMOVE_VOWELSBELOW_THRESHOLDADDSAME_CHARSFIBCORRECT_BRACKETINGMONOTONICCOMMONLARGEST_PRIME_FACTORSUM_TO_NDERIVATIVEFIBFIBVOWELS_COUNTCIRCULAR_SHIFTDIGITSUMFRUIT_DISTRIBUTIONPLUCKSEARCHSTRANGE_SORT_LISTWILL_IT_FLYSMALLEST_CHANGETOTAL_MATCHIS_MULTIPLY_PRIMEIS_SIMPLE_POWERIS_CUBEHEX_KEYDECIMAL_TO_BINARYIS_HAPPYNUMERICAL_LETTER_GRADEPRIME_LENGTHSTARTS_ONE_ENDSSOLVEANTI_SHUFFLEGET_ROWSORT_ARRAYENCRYPTNEXT_SMALLESTIS_BOREDANY_INTENCODESKJKASDKDCHECK_DICT_CASECOUNT_UP_TOMULTIPLYCOUNT_UPPERCLOSEST_INTEGERMAKE_A_PILEWORDS_STRINGCHOOSE_NUMROUNDED_AVGUNIQUE_DIGITSBY_LENGTHEVEN_ODD_PALINDROMECOUNT_NUMSMOVE_ONE_BALLEXCHANGEHISTOGRAMREVERSE_DELETEODD_COUNTMINSUBARRAYSUMMAX_FILLSELECT_WORDSGET_CLOSEST_VOWELMATCH_PARENSMAXIMUMSOLUTIONADD_ELEMENTSGET_ODD_COLLATZVALID_DATESPLIT_WORDSIS_SORTEDINTERSECTIONPROD_SIGNSMINPATHTRIDIGITSIS_NESTEDSUM_SQUARESCHECK_IF_LAST_CHAR_IS_A_LETTERCAN_ARRANGELARGEST_SMALLEST_INTEGERSCOMPARE_ONEIS_EQUAL_TO_SUM_EVENSPECIAL_FACTORIALFIX_SPACESFILE_NAME_CHECKWORDS_IN_SENTENCESIMPLIFYORDER_BY_POINTSSPECIALFILTERGET_MAX_TRIPLESBFSORTED_LIST_SUMX_OR_YDOUBLE_THE_DIFFERENCECOMPARESTRONGEST_EXTENSIONCYCPATTERN_CHECKEVEN_ODD_COUNTINT_TO_MINI_ROMANRIGHT_ANGLE_TRIANGLEFIND_MAXEATDO_ALGEBRASTRING_TO_MD5GENERATE_INTEGERS