**105 scoring-rules questions.**

Context
We have a register of people in our database, with a record of teams and leaders. Every day we send out a survey to each team asking them to rate their leader on whether they performed a ...

I am wanting to weight the continuous rank probability scoring rule so I can focus on the left side of the tail. Could someone please tell me how I would do this? I don't need code as such, just an ...

I'm cross validating a model that's trying to predict a count. If this was a binary classification problem, I'd calculate out-of-fold AUC, and if this was a regression problem I'd calculate out-of-...

I have a question for which i requrest your comments and recommendations:
In my research, I use a National level entrance test, which has high signficance for the students in their pursuit of higher ...

I have built two regression models to predict sales of different products based on a number of explanatory variables, with an offset term for the number of days each product was on sale. One is a ...

I would like to get some p-values for predictions in an independent test set according to proper scoring rules. So I would like to know if my brier score/logarithmic score is statistically ...

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference.
...

I need to predict (estimate) probabilities of (rare) events when the training data only contains the yes/no indicator.
I.e., my target (dependent) variable is binary (logical).
What I need is not ...

For an OLS model the mean squared error can be used to assess the fit of the trained model on the validation data.
What is the equivalent for a logistic regression model? Can I simply use the ...

I want to use a custom objective function with xgboost: 1 - (log(y) - log(p)) / (log(y) - log(q))y = true value, p = my probabilities, q = some other base ...

I have built a quasi-Poisson regression to predict sales of different products based on a number of explanatory variables, with an offset term for the number of days each product was on sale.
To ...

I recently completed a Kaggle competition in which roc auc score was used as per competition requirement. Before this project, I normally used f1 score as the metric to measure model performance. ...

I tried to solve the problem of comparing two estimators for soccer matches. The "estimators" are actually two punters trying two predict games results. The predicted value is between 0 and 1. The ...

I have recently been learning about proper scoring rules for probabilistic classifiers. Several threads on this website have made a point of emphasizing that accuracy is an improper scoring rule and ...

Several sources state that the score function for the likelihood of a cox model is
$$
\dfrac{\partial{}l(\beta)}{\partial\beta}=\Big(X_{i}\delta_i^T-\sum\limits_{i=1}^{n}\delta_i\dfrac{\sum\limits_{j\...

A proper scoring rule is a rule that is maximized by a 'true' model and it doesn't allow 'hedging' or gaming the system (deliberately reporting different results as is the true belief of the model to ...

When we search for a numerical way to find $\hat{\beta}$ in a GLM (say, a logistic regression), we could do a numerical optimization (minimization) of the negative log-likelihood.
But instead, we go ...

I need help with figuring out a proper scoring rule for the following task.
There are 11 possible outcomes.
There is a true probability distribution over these outcomes (known to me but not you). ...

I have seen different papers use different terms to express the scoring rules that they used to compare Bayesian models. Some of those terms are,
Log Predictive Density (Bayesian Data Analysis - by ...

The situation:
I have a logistic model that should predict a defect (1=defect, 0=no defect). My model uses 4 out of 14 parameters, which are significant for my ...

I would like to evaluate the performance of my machine learning model on a test set, but I only have access to the ground truth for a subset of the test set, say 60% of it. On the remaining 40%, the ...

I am building a machine learning model to attempt to predict the winner of a sports match based on historical statistics of the two teams.
My model (a neural network) appears to get about 70% ...

[I'm not a mathematician, so please forgive any misuse of terminology]
One way of understanding scoring rules is that they measure the 'distance' between the truth value of a statement, and the ...

My problem concerns an aptitude test containing a set of single choice items ($x_1 x_2 .. x_n$). For each item, the participant may have selected option 1 to 5. These choices are scored dichotomously (...

I'm using scikit package with RandomForestClassifier, trying to predict binary or multi-lable classifications.
I'm looking for a way to estimate the reliability of the model but really can't figure ...

I am trying to prepare a questionnaire of 30 items in order to assess the quality of published papers. Based on the importance of each criterion, I want to assign weights. Should I assign the weights ...

I need to create a scorecard where I can compare and contrast 3 different managers on one single measures/metric. The problem I'm having is to come up with a defend-able and fair scorecard logic/...

Background: There are some great questions/answers here on how to calibrate models which predict probabilities of an outcome happening. For example
Brier score, and its decomposition into resolution, ...

Dr Frank Harrell mentioned in his book and BIOS 330 course that
Accuracy score used to drive model building should be a continuous score that utilizes all the information in the data (e.g. Brier ...

I am running an analysis on the probability of loan default using logistic regression and random forests.
When I use logistic regression, the prediction is always all '1' (which means good loan). ...

I wonder what is the best statistical method to validate/score/evaluate a regression Neural Network used to predict probabilities (an example would be using a Regression NN to predict the probability ...

I've been studying (and applying) SVMs for some time now, mostly through kernlab in R.
...

Merkle & Steyvers (2013) write:
To formally define a proper scoring rule, let $f$ be a probabilistic
forecast of a Bernoulli trial $d$ with true success probability $p$.
Proper scoring ...

Suppose a populous nation has a high homicide rate and an understaffed police force. The police chief hires a statistician and together they decide to take a preventative approach by identifying ...

I'm testing several model that predict a binary variable. I've applied the logarithmic, Brier, and spherical scoring rules. Now, how do I determine what is a meaningful difference between the scores ...

Next week I will teach my students the score function and its variance (i.e.: fisher information).
I am looking for way(s) to illustrate these concepts so to help my students understand them (and not ...

I read about using scoring rules to evaluate the performance of predictive models. In the Wikipedia article about the Brier score, it is stated:
The Brier score is appropriate for binary and ...

The Spherical scoring rule is known to be strictly proper.
However, it is not very intuitive.
It's arccos, however, is the angle between the prediction ...

Suppose to have a bivariate variable $z_t=(x_t, y_t)$ indexed by $t=1,2, ..., T$. Suppose now that the two components have different support, i.e. in my specific problem $x_t \in \mathcal{S}$, where $\...

A very popular index to measure the stability of characteristics of a scorecard is defined by the following formula:
$$SI = \frac{1}{n} \sum \text{(actual in %)-(expected in %)} \cdot \log\left(\frac{...

I need to learn a Bayesian Network Structure from a dataset. I read the book titled "Learning Bayesian Networks" written Neapolitan and Richard but I have no clear idea.
According to the book from ...

- logistic
- classification
- forecasting
- predictive-models
- validation
- r
- regression
- machine-learning
- probability
- accuracy
- model-evaluation
- maximum-likelihood
- prediction
- error
- calibration
- neural-networks
- model-selection
- cox-model
- loss-functions
- hypothesis-testing
- cross-validation
- python
- likelihood
- model-comparison
- bayesian-network

- How is graphics RAM different from system RAM?
- Algebra 2 textbooks that incorrectly claim that all solutions of polynomial equations can be found
- How to get the maximum attachment size of a mail server which perform ip connection whitelisting?
- Should I send professors 10 dollars for illegally downloading their books?
- Number of transformations until repeat
- Man running icon
- If someone wanted to pretend to be Satoshi by posting a fake signature to defraud people how could they?
- Does this cantrip have a fair risk/reward balance?
- Continuous power between multi-gang boxes with 3 way switches
- Why is the EU concerned about the UK "unilaterally withdrawing" from a proposed Irish backstop?
- How to draw the following matrix in latex?
- My DNS record can only point to an IP address how to make it reach for a port?
- Is there a difference between terracotta and earthenware?
- Science fiction series where in each novel, a man travels from planet to planet and helps the inhabitants in overthrowing their oppressors
- Is it ethical to use knowledge in main job for side gig?
- What is a blind encoder?
- The Book of Why by Judea Pearl: Why is he bashing statistics?
- How fast is the Internet on board Iceland Air flights?
- Which passport number should I use?
- Has women's suffrage ever decided an election?
- Could people of the 14th century create an inline engine for use in aircraft and vehicles?
- Is it ever okay to use lists in a relational database?
- 2 x 2 lines equation on a single line separated by a text in the middle
- Are most cases of polio caused by polio vaccine?