# precision's questions - English 1answer

297 precision questions.

### 3 Calculations after NumberForm [duplicate]

1 answers, 228 views precision number-representation
As specified in documentation center the functions AccountingForm, NumberForm, etc. are useful only to display numbers in ...

### equal expressions give different numerical result

I have two expressions, and they are really equal: ...

### 3 How to improve resolution of plot with underflow and overflow?

I am working with a function that has the formExp[-a^2](Erfi[a]-Log[-1/a]-Log[a])/awhere a is complex number. When I tried ...

### 2 How to solve an inequality using low precision (< MachinePrecision)?

I would like to use low precision data (based on measurement results with a low precision) to solve a set of inequalities. I have prepared a minimum working example to demonstrate the issue. I am ...

### 12 Keep Round-Off errors for educational purpose

I want to show examples of round-off errors in some numerical algorithms to my student, in order to motivate the study of algorithms with a better behavior. While it is easy in any other language, I ...

### 7 How does Mathematica determine when to use scientific notation?

I have code that takes an initial guess for a set of parameters to use in a function that I want to minimize, calculates the function, prints the value, recalculates new parameters via the gradient ...

### 6 How to create an Interval object with specific end points?

3 answers, 110 views precision intervals
When creating an Interval object, the end points are automatically widened when an end point is an inexact number. For instance: ...

### 5 Integrating a bivariate distribution over a region bounded by a straight line

Summary of problem: I'm using Mathematica version 8 to try to integrate the bivariate distribution over a region bounded by a straight line. The two random variables are uncorrelated. When I use ...

### 1 Diagnosing the numerical stability of a function

2 answers, 33 views precision accuracy stability
I have programmed a function f[x] that seems to be very unstable numerically. More precisely, I noticed that for certain arguments in arbitrary precision, the ...

### 3 Arbitrary Precision, Nearly-Singular Matrices, and LinearSolve

I have been trying to solve a nonlinear eigenvalue problem, $\mathbf{M}(\lambda) \mathbf{v} = 0$, in Mathematica using Newton's method. The core of the algorithm relies upon an inverse iteration, ...

### 5 DiscretePlot Giving Incorrect Answer

I know similar questions have been asked before about bugs in Integrate, but I'm not sure what the particular problem is here (and whether or not there is indeed a bug). When I perform the integral: ...

### 5 Why does N not upgrade precision? [duplicate]

Precision[N[1.0, 20]] Precision[N[1, 20]]MachinePrecision 20.It would be so much more intuitive and less error prone, if <...

### 5 Early vs. late application of arbitrary precision

The following arbitrary-precision computations, in which arbitrary precision is applied early (to the inputs), both work as expected: ...

### How to correct not precise values

1 answers, 42 views list-manipulation import precision
I have the source file file = "test.txt" with these values: ...

### FullForm[1. - (0.3 + 0.4)] gives 0.30000000000000004` [duplicate]

0 answers, 55 views error precision
I had a serious error propagation in my computations (Mathematica 10.4.1 Windows 64 bit). After some investigation, I realized the problem comes from something like this: FullForm[1. - (0.3 + 0.4)] ...

### 1 Rounding numbers to certain numbers of significant figures

2 answers, 61 views precision output
I have code which outputs numbers, for example,(0.2945847, 0.8647834, 1.6*10^-6) and similar. I would like to take these numbers and round them to 4 significant figures, but I can't find a way to do ...

### Identical expressions subtracted from each other giving small but significant errors

0 answers, 29 views list-manipulation precision
In my current code, I have formed a list of numbers with two different methods. For the first method, I wrote ...

### 2 How can I remove the FindRoot error from my calculations?

3 answers, 240 views plotting equation-solving precision
If you run my calculations given below ...

### Precision for function

I have a notebook where I have to work with high precision (say, 100). I have tried several methods to define it for the entire notebook, including ...

### Computing a sum with high precision

2 answers, 58 views numerics precision numerical-value
I am aware that a similar question has been asked, for example, here and here, but none of the proposed solutions solves my problem, so please wait before you mark this question as a duplicate. ...

### 1 Performance Tuning - How can I make Mathematica to use less than certain digits number?

I would like to know if there is a way to make Mathematica to use at most certain digit number. For example, Suppose I want to calculate 2*Pi.x=N[Pi] y=2*x...

### 3 SetAccuracy not resulting in duplicates for GeoData like SetPrecision does

1 answers, 33 views precision accuracy maps
I am continuing to work with GPS data related to trail routes and am trying to optimize the code by removing route points from a GPX file that are very close to each other by setting the accuracy of ...

### 1 Precision error in FindMinimum

I am trying to solve a system of three non-linear equations for three unknowns (e1, e2, phi). I am using FindMinimum to solve the system (I am minimizing the norm ...

### 3 MinRecursion changes result of integration by 100 orders of magnitude

1 answers, 66 views numerical-integration precision
I want to compute a triple integral. I did it using two codes: the first is ...

### 5 Precision-related error in Audio

1 answers, 66 views bugs precision sound machine-precision
Bug introduced in 11.3 or earlier CASE:4075745 I have been experimenting with using Audio and associated functions for data acquisition using an external sound ...

### 3 How to avoid density plot being too ragged?

1 answers, 58 views plotting precision
I have a density plot as below:-v = {2, 3}; w = {4, 6}; DensityPlot[Exp[-w.Abs[v - {x, y}]], {x, 0, 5}, {y, 0, 7}]As you can see, the boundary of the white ...

### 1 How does one specify numerical precision for FortranForm[] and CForm[]

1 answers, 46 views output-formatting precision
I have a long function composed of polynomials and exponentials that I would like to convert to an appropriate form to cut and paste as an equation into SolidWorks. FortranForm gets me close and with ...

### 3 WorkingPrecision in NDSolve causes failure when solving a simple PDE

I have fivefold multiple integral and I wanted a speed calculations. I came across on this Question and had already begun the problems. Here is a toy example with simple double integral: ...

### 4 SetPrecision and setting precision with a backtick

3 answers, 100 views numerics precision
When you use SetPrecision to set the precision of say 1.1 you will get some seemingly random numbers padded before the rest will ...

### Eigenvalues not invariant after similarity transformation

I have two examples of matrices that undergo a similarity transformation: a random matrix, and an exclusion-process inspired matrix. The similarity transformation $\sigma$ is given by an $L \times L$ ...

### 4 How to force precision in region plots containing Erfc

2 answers, 147 views plotting numerics precision
I'm trying to use regionplot for functions that involve Erfc. A slightly contrived example follows, but it illustrates my difficulties. The command ...

### 1 How can I improve the numerical results while working Mathematica

2 answers, 55 views equation-solving numerics precision
I'm trying to solve nonlinear equations using Newton-type methods with very high accuracy using Mathematica. I found many research papers in which the numerical results are calculated with very high ...

### 1 Numerical error in compile, Strange behaviour in numerical evaluation

First of all, apologies for the large numbers, I couldn't find a working example for a far lower order and the zeros are necessary... my problem is this: I have a large polynomial (from high-order ...

### 8 Contour integration, residues and precision of the poles

1 answers, 145 views calculus-and-analysis complex precision
Task I am trying to evaluate the contour integral of some functions. To have a concrete example, let's use $$f(z,s) = \frac{1+(4+2s)\, z}{z - \left( 9 + 35s + 24s^2 + 4s^3 \right) z^2 + 8z^3}$$<...

### 6 How to get accurate solutions from FindRoot?

2 answers, 263 views equation-solving precision

### 5 What is a sufficient decrease in the merit function?

1 answers, 273 views equation-solving error precision
As far as I understand, the merit function measures how far the data is from the fitted function. I am calulcating the second partial derivative of a function, and then numerically finding its maxima ...

### 5 Precision loss in FourierDCT

2 answers, 197 views fourier-analysis precision
I have high precision data that I want to do Fourier transform with, but it looks like the precision is lost dramatically. ...

### Precision issue using functions

1 answers, 29 views functions precision
I just want to get a smooth plot but the calculated values are "discrete" which seems to be a precision problem. I tried N[],SetPrecision[],SetAccuary[] without any success. So I paste the Code in its ...

### Iterative calculation of a number-theoretical constant [closed]

2 answers, 117 views precision number-theory
Recently the decimal expansion of a number theoretic constant was searched for, which is the analog of the Landau-Ramanujan constant in a certain context. The constant starts 0.638909... It can be ...

### 1 Interpolation is not smooth

Consider a function ...

### 1 Printing out reals in a table with a specified number of digits

I would like to print out an list of reals in a table with perfectly identical number of digits so that they can be perfectly right-aligned. However, once I convert them to strings, ...

### 1 Plotting with small variations along the vertical axis

3 answers, 66 views plotting precision
I can't see the small changes of my function when I plot it. What can I do? ...

### 3 digits of NDSolveValue

I am solving PDE by means of NDSolve and I am interested in a highly accurate value for the solution at a certain point. For some reason, the solution obtained by ...

### High precision NDSolve

0 answers, 58 views differential-equations precision
I need to solve an ODE with very high precision. I am not sure yet how much precision I need so I have some question before changing the settings. From what I understand, the working precision is how ...

### 2 How to avoid problems with the large imaginary part of LegendreQ?

I am trying to evaluate sums involving LegendreQ[n,x], the n-th order Legendre function of the second kind $Q_n(x)$, at large $x>1$. However, the problem is that ...

### 2 WorkingPrecision and AccuracyGoal for using with NIntegrate

1 answers, 1.200 views numerical-integration precision accuracy
I'm modeling a quantum tunneling barrier. Because I want to express my quantities in mks units, that means that some of the variables I'm using are pretty small (10^-9). I use NIntegrate for part of ...

### 6 Mathematica precision common problems

1 answers, 141 views precision arbitrary-precision accuracy
When I do something a little bit more complicated than standard documentation examples, I often hit precision problem. Or I accidentally disprove Riemann hypothesis Like this ...

### 5 Is FindRoot wrong about its WorkingPrecision?

2 answers, 516 views equation-solving precision
EDIT: I filed a bug report and after a small back-and-forth the support person agreed this is a bug. He said: "I have filed a report with the developers to issue warning messages when the arithmetic ...